Image-based airway analysis and mapping

ABSTRACT

Navigation of an instrument within a luminal network can include image-based airway analysis and mapping. Image-based airway analysis can include detecting one or more airways in an image captured within a luminal network and determining branching information indicative of how the current airway in which the image is captured branches into the detected “child” airways. Image-based airway mapping can include mapping the one or more detected airways to corresponding expected airways of the luminal network in the preoperative model.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefit of U.S. Provisional Application No.62/678,881, filed May 31, 2018, which is hereby incorporated byreference in its entirety.

TECHNICAL FIELD

This disclosure relates generally to systems and methods for navigationof medical instruments, and more particularly to image-based airwayanalysis and mapping for navigating robotically-controlled medicalinstruments.

BACKGROUND

Medical procedures such as endoscopy (e.g., bronchoscopy) may involveaccessing and visualizing the inside of a patient's lumen (e.g.,airways) for diagnostic and/or therapeutic purposes. During a procedure,a flexible tubular tool or instrument, such as an endoscope, may beinserted into the patient's body. In some instances, a second instrumentcan be passed through the endoscope to a tissue site identified fordiagnosis and/or treatment.

Bronchoscopy is a medical procedure that allows a physician to examinethe inside conditions of airways in a patient's lungs, such as bronchiand bronchioles. During the medical procedure, a thin, flexible tubulartool or instrument, known as a bronchoscope, may be inserted into thepatient's mouth and passed down the patient's throat into his or herlung airways towards a tissue site identified for subsequent diagnosisand/or treatment. The bronchoscope can have an interior lumen (a“working channel”) providing a pathway to the tissue site, and cathetersand various medical tools can be inserted through the working channel tothe tissue site.

In certain medical procedures, surgical robotic systems may be used tocontrol the insertion and/or manipulation of the surgical tools.Surgical robotic systems may include at least one robotic arm or otherinstrument positioning device including a manipulator assembly used tocontrol the positioning of the surgical tool during the procedures.

SUMMARY

Robotically-enabled medical systems can be used to perform a variety ofmedical procedures, including both minimally invasive procedures, suchas laparoscopic procedures, and non-invasive procedures, such asendoscopic procedures. Among endoscopic procedures, robotically-enabledmedical systems can be used to perform bronchoscopy, ureteroscopy,gastroenterology, etc. During such procedures, a physician and/orcomputer system can navigate a medical instrument through a luminalnetwork of a patient. The luminal network can include a plurality ofbranched lumens (such as in bronchial or renal networks), or a singlelumen (such as a gastrointestinal tract). The robotically-enabledmedical systems can include navigation systems for guiding (or assistingwith the guidance of) the medical instrument through the luminalnetwork.

Embodiments of this disclosure relate to systems and techniques forimage-based airway analysis and mapping. Image-based airway analysis andmapping may aid navigation through the luminal network. Image-basedairway analysis can include identifying, within an image captured withan imaging device on the instrument, one or more airways associated withone or more branches of a luminal network and determining branchinginformation indicative of how the current airway in which the image iscaptured branches into the detected “child” airways. Image-based airwaymapping can include mapping the identified airways to correspondingbranches of the luminal network. These systems and techniques may beused to determine or estimate the position of an instrument within theluminal network. The systems, methods, and devices of this disclosureeach have several innovative aspects, no single one of which is solelyresponsible for the desirable attributes disclosed herein.

In one aspect, there is provided a method of navigating an instrumentthrough a luminal network, the method comprising: capturing a pluralityof images within the luminal network with an imaging device positionedon the instrument, the plurality of images comprising at least a firstimage captured at a first time and a second image captured at a secondtime subsequent to the first time; identifying a first airway in thefirst image; identifying two or more airways in the second image;determining, based on the first airway in the first image and the two ormore airways in the second image, that an overlap condition is met;accessing preoperative model data indicative of an expected count ofairways corresponding to a location of the instrument during the secondtime; and determining, based on the preoperative model data and thedetermination that the overlap condition is met, a mapping between thetwo or more airways in the second image and the airways in thepreoperative model data.

In another aspect, there is provided a non-transitory computer readablestorage medium having stored thereon instructions that, when executed,cause a processor of a device to at least: capture a plurality of imageswithin a luminal network with an imaging device positioned on aninstrument, the plurality of images comprising at least a first imagecaptured at a first time and a second image captured at a second timesubsequent to the first time; identify a first airway in the firstimage; identify two or more airways in the second image; determine,based on the first airway in the first image and the two or more airwaysin the second image, that an overlap condition is met; accesspreoperative model data indicative of an expected count of airwayscorresponding to a location of the instrument during the second time;and determine, based on the preoperative model data and thedetermination that the overlap condition is met, a mapping between thetwo or more airways in the second image and the airways in thepreoperative model data.

In yet another aspect, there is provided a robotic surgical system formapping one or more airways in a luminal network, the system comprising:an instrument having: an elongate body configured to be inserted intothe luminal network, and an imaging device positioned on a distalportion of the elongate body; an instrument positioning device attachedto the instrument, the instrument positioning device configured to movethe instrument through the luminal network; at least onecomputer-readable memory having stored thereon executable instructions;and one or more processors in communication with the at least onecomputer-readable memory and configured to execute the instructions tocause the system to at least: capture a plurality of images within theluminal network with an imaging device positioned on the instrument, theplurality of images comprising at least a first image captured at afirst time and a second image captured at a second time subsequent tothe first time; identify a first airway in the first image; identify twoor more airways in the second image; determine, based on the firstairway in the first image and the two or more airways in the secondimage, that an overlap condition is met; access preoperative model dataindicative of an expected count of airways corresponding to a locationof the instrument during the second time; and determine, based on thepreoperative model data and the determination that the overlap conditionis met, a mapping between the two or more airways in the second imageand the airways in the preoperative model data.

BRIEF DESCRIPTION OF THE DRAWINGS

The disclosed aspects will hereinafter be described in conjunction withthe appended drawings, provided to illustrate and not to limit thedisclosed aspects, wherein like designations denote like elements.

FIG. 1 illustrates an embodiment of a cart-based robotic system arrangedfor diagnostic and/or therapeutic bronchoscopy procedure(s).

FIG. 2 depicts further aspects of the robotic system of FIG. 1.

FIG. 3 illustrates an embodiment of the robotic system of FIG. 1arranged for ureteroscopy.

FIG. 4 illustrates an embodiment of the robotic system of FIG. 1arranged for a vascular procedure.

FIG. 5 illustrates an embodiment of a table-based robotic systemarranged for a bronchoscopy procedure.

FIG. 6 provides an alternative view of the robotic system of FIG. 5.

FIG. 7 illustrates an example system configured to stow robotic arm(s).

FIG. 8 illustrates an embodiment of a table-based robotic systemconfigured for a ureteroscopy procedure.

FIG. 9 illustrates an embodiment of a table-based robotic systemconfigured for a laparoscopic procedure.

FIG. 10 illustrates an embodiment of the table-based robotic system ofFIGS. 5-9 with pitch or tilt adjustment.

FIG. 11 provides a detailed illustration of the interface between thetable and the column of the table-based robotic system of FIGS. 5-10.

FIG. 12 illustrates an exemplary instrument driver.

FIG. 13 illustrates an exemplary medical instrument with a pairedinstrument driver.

FIG. 14 illustrates an alternative design for an instrument driver andinstrument where the axes of the drive units are parallel to the axis ofthe elongated shaft of the instrument.

FIG. 15 depicts a block diagram illustrating a localization system thatestimates a location of one or more elements of the robotic systems ofFIGS. 1-10, such as the location of the instrument of FIGS. 13-14, inaccordance to an example embodiment.

FIG. 16 illustrates an example of an instrument navigating a luminalnetwork.

FIG. 17 illustrates an example command console for arobotically-controlled surgical system.

FIG. 18 illustrates a distal end of an embodiment of a medicalinstrument.

FIG. 19 depicts a flowchart illustrating an example method forimage-based airway analysis and mapping.

FIG. 20 illustrates an example image of an interior of a branch of aluminal network.

FIG. 21 illustrates a simplified view of a luminal network.

FIG. 22 illustrates two different example airway detection results.

FIG. 23A illustrates an airway analysis and mapping system according toexample embodiments.

FIG. 23B illustrates the temporal context in which the image-basedairway analysis according to example embodiments is performed.

FIG. 24 illustrates the spatial nature of the image-based airwayanalysis according to example embodiments.

FIG. 25 illustrates additional example relationships exhibited bydetected airways in two consecutive images.

FIG. 26 depicts a flowchart illustrating an example method forimage-based airway analysis and mapping.

DETAILED DESCRIPTION I. Overview

Aspects of the present disclosure may be integrated into arobotically-enabled medical system capable of performing a variety ofmedical procedures, including both minimally invasive, such aslaparoscopy, and non-invasive, such as endoscopy, procedures. Amongendoscopy procedures, the system may be capable of performingbronchoscopy, ureteroscopy, gastroscopy, etc.

In addition to performing the breadth of procedures, the system mayprovide additional benefits, such as enhanced imaging and guidance toassist the physician. Additionally, the system may provide the physicianwith the ability to perform the procedure from an ergonomic positionwithout the need for awkward arm motions and positions. Still further,the system may provide the physician with the ability to perform theprocedure with improved ease of use such that one or more of theinstruments of the system can be controlled by a single user.

Various embodiments will be described below in conjunction with thedrawings for purposes of illustration. It should be appreciated thatmany other implementations of the disclosed concepts are possible, andvarious advantages can be achieved with the disclosed implementations.Headings are included herein for reference and to aid in locatingvarious sections. These headings are not intended to limit the scope ofthe concepts described with respect thereto. Such concepts may haveapplicability throughout the entire specification.

A. Robotic System—Cart.

The robotically-enabled medical system may be configured in a variety ofways depending on the particular procedure. FIG. 1 illustrates anembodiment of a cart-based robotically-enabled system 10 arranged for adiagnostic and/or therapeutic bronchoscopy procedure. During abronchoscopy, the system 10 may comprise a cart 11 having one or morerobotic arms 12 to deliver a medical instrument, such as a steerableendoscope 13, which may be a procedure-specific bronchoscope forbronchoscopy, to a natural orifice access point (i.e., the mouth of thepatient positioned on a table in the present example) to deliverdiagnostic and/or therapeutic tools. As shown, the cart 11 may bepositioned proximate to the patient's upper torso in order to provideaccess to the access point. Similarly, the robotic arms 12 may beactuated to position the bronchoscope relative to the access point. Thearrangement in FIG. 1 may also be utilized when performing agastro-intestinal (GI) procedure with a gastroscope, a specializedendoscope for GI procedures. FIG. 2 depicts an example embodiment of thecart in greater detail.

With continued reference to FIG. 1, once the cart 11 is properlypositioned, the robotic arms 12 may insert the steerable endoscope 13into the patient robotically, manually, or a combination thereof. Asshown, the steerable endoscope 13 may comprise at least two telescopingparts, such as an inner leader portion and an outer sheath portion, eachportion coupled to a separate instrument driver from the set ofinstrument drivers 28, each instrument driver coupled to the distal endof an individual robotic arm. This linear arrangement of the instrumentdrivers 28, which facilitates coaxially aligning the leader portion withthe sheath portion, creates a “virtual rail” 29 that may be repositionedin space by manipulating the one or more robotic arms 12 into differentangles and/or positions. The virtual rails described herein are depictedin the Figures using dashed lines, and accordingly the dashed lines donot depict any physical structure of the system. Translation of theinstrument drivers 28 along the virtual rail 29 telescopes the innerleader portion relative to the outer sheath portion or advances orretracts the endoscope 13 from the patient. The angle of the virtualrail 29 may be adjusted, translated, and pivoted based on clinicalapplication or physician preference. For example, in bronchoscopy, theangle and position of the virtual rail 29 as shown represents acompromise between providing physician access to the endoscope 13 whileminimizing friction that results from bending the endoscope 13 into thepatient's mouth.

The endoscope 13 may be directed down the patient's trachea and lungsafter insertion using precise commands from the robotic system untilreaching the target destination or operative site. In order to enhancenavigation through the patient's lung network and/or reach the desiredtarget, the endoscope 13 may be manipulated to telescopically extend theinner leader portion from the outer sheath portion to obtain enhancedarticulation and greater bend radius. The use of separate instrumentdrivers 28 also allows the leader portion and sheath portion to bedriven independently of each other.

For example, the endoscope 13 may be directed to deliver a biopsy needleto a target, such as, for example, a lesion or nodule within the lungsof a patient. The needle may be deployed down a working channel thatruns the length of the endoscope to obtain a tissue sample to beanalyzed by a pathologist. Depending on the pathology results,additional tools may be deployed down the working channel of theendoscope for additional biopsies. After identifying a nodule to bemalignant, the endoscope 13 may endoscopically deliver tools to resectthe potentially cancerous tissue. In some instances, diagnostic andtherapeutic treatments can be delivered in separate procedures. In thosecircumstances, the endoscope 13 may also be used to deliver a fiducialto “mark” the location of the target nodule as well. In other instances,diagnostic and therapeutic treatments may be delivered during the sameprocedure.

The system 10 may also include a movable tower 30, which may beconnected via support cables to the cart 11 to provide support forcontrols, electronics, fluidics, optics, sensors, and/or power to thecart 11. Placing such functionality in the tower 30 allows for a smallerform factor cart 11 that may be more easily adjusted and/orre-positioned by an operating physician and his/her staff. Additionally,the division of functionality between the cart/table and the supporttower 30 reduces operating room clutter and facilitates improvingclinical workflow. While the cart 11 may be positioned close to thepatient, the tower 30 may be stowed in a remote location to stay out ofthe way during a procedure.

In support of the robotic systems described above, the tower 30 mayinclude component(s) of a computer-based control system that storescomputer program instructions, for example, within a non-transitorycomputer-readable storage medium such as a persistent magnetic storagedrive, solid state drive, etc. The execution of those instructions,whether the execution occurs in the tower 30 or the cart 11, may controlthe entire system or sub-system(s) thereof. For example, when executedby a processor of the computer system, the instructions may cause thecomponents of the robotics system to actuate the relevant carriages andarm mounts, actuate the robotics arms, and control the medicalinstruments. For example, in response to receiving the control signal,the motors in the joints of the robotics arms may position the arms intoa certain posture.

The tower 30 may also include a pump, flow meter, valve control, and/orfluid access in order to provide controlled irrigation and aspirationcapabilities to the system that may be deployed through the endoscope13. These components may also be controlled using the computer system ofthe tower 30. In some embodiments, irrigation and aspirationcapabilities may be delivered directly to the endoscope 13 throughseparate cable(s).

The tower 30 may include a voltage and surge protector designed toprovide filtered and protected electrical power to the cart 11, therebyavoiding placement of a power transformer and other auxiliary powercomponents in the cart 11, resulting in a smaller, more moveable cart11.

The tower 30 may also include support equipment for the sensors deployedthroughout the robotic system 10. For example, the tower 30 may includeoptoelectronics equipment for detecting, receiving, and processing datareceived from the optical sensors or cameras throughout the roboticsystem 10. In combination with the control system, such optoelectronicsequipment may be used to generate real-time images for display in anynumber of consoles deployed throughout the system, including in thetower 30. Similarly, the tower 30 may also include an electronicsubsystem for receiving and processing signals received from deployedelectromagnetic (EM) sensors. The tower 30 may also be used to house andposition an EM field generator for detection by EM sensors in or on themedical instrument.

The tower 30 may also include a console 31 in addition to other consolesavailable in the rest of the system, e.g., console mounted on top of thecart. The console 31 may include a user interface and a display screen,such as a touchscreen, for the physician operator. Consoles in thesystem 10 are generally designed to provide both robotic controls aswell as preoperative and real-time information of the procedure, such asnavigational and localization information of the endoscope 13. When theconsole 31 is not the only console available to the physician, it may beused by a second operator, such as a nurse, to monitor the health orvitals of the patient and the operation of the system 10, as well as toprovide procedure-specific data, such as navigational and localizationinformation. In other embodiments, the console 30 is housed in a bodythat is separate from the tower 30.

The tower 30 may be coupled to the cart 11 and the endoscope 13 throughone or more cables or connections (not shown). In some embodiments, thesupport functionality from the tower 30 may be provided through a singlecable to the cart 11, simplifying and de-cluttering the operating room.In other embodiments, specific functionality may be coupled in separatecabling and connections. For example, while power may be providedthrough a single power cable to the cart 11, the support for controls,optics, fluidics, and/or navigation may be provided through a separatecable.

FIG. 2 provides a detailed illustration of an embodiment of the cart 11from the cart-based robotically-enabled system shown in FIG. 1. The cart11 generally includes an elongated support structure 14 (often referredto as a “column”), a cart base 15, and a console 16 at the top of thecolumn 14. The column 14 may include one or more carriages, such as acarriage 17 (alternatively “arm support”) for supporting the deploymentof one or more robotic arms 12 (three shown in FIG. 2). The carriage 17may include individually configurable arm mounts that rotate along aperpendicular axis to adjust the base of the robotic arms 12 for betterpositioning relative to the patient. The carriage 17 also includes acarriage interface 19 that allows the carriage 17 to verticallytranslate along the column 14.

The carriage interface 19 is connected to the column 14 through slots,such as slot 20, that are positioned on opposite sides of the column 14to guide the vertical translation of the carriage 17. The slot 20contains a vertical translation interface to position and hold thecarriage 17 at various vertical heights relative to the cart base 15.Vertical translation of the carriage 17 allows the cart 11 to adjust thereach of the robotic arms 12 to meet a variety of table heights, patientsizes, and physician preferences. Similarly, the individuallyconfigurable arm mounts on the carriage 17 allow the robotic arm base 21of the robotic arms 12 to be angled in a variety of configurations.

In some embodiments, the slot 20 may be supplemented with slot coversthat are flush and parallel to the slot surface to prevent dirt andfluid ingress into the internal chambers of the column 14 and thevertical translation interface as the carriage 17 vertically translates.The slot covers may be deployed through pairs of spring spoolspositioned near the vertical top and bottom of the slot 20. The coversare coiled within the spools until deployed to extend and retract fromtheir coiled state as the carriage 17 vertically translates up and down.The spring-loading of the spools provides force to retract the coverinto a spool when the carriage 17 translates towards the spool, whilealso maintaining a tight seal when the carriage 17 translates away fromthe spool. The covers may be connected to the carriage 17 using, forexample, brackets in the carriage interface 19 to ensure properextension and retraction of the cover as the carriage 17 translates.

The column 14 may internally comprise mechanisms, such as gears andmotors, that are designed to use a vertically aligned lead screw totranslate the carriage 17 in a mechanized fashion in response to controlsignals generated in response to user inputs, e.g., inputs from theconsole 16.

The robotic arms 12 may generally comprise robotic arm bases 21 and endeffectors 22, separated by a series of linkages 23 that are connected bya series of joints 24, each joint comprising an independent actuator,each actuator comprising an independently controllable motor. Eachindependently controllable joint represents an independent degree offreedom available to the robotic arm 12. Each of the robotic arms 12 mayhave seven joints, and thus provide seven degrees of freedom. Amultitude of joints result in a multitude of degrees of freedom,allowing for “redundant” degrees of freedom. Having redundant degrees offreedom allows the robotic arms 12 to position their respective endeffectors 22 at a specific position, orientation, and trajectory inspace using different linkage positions and joint angles. This allowsfor the system to position and direct a medical instrument from adesired point in space while allowing the physician to move the armjoints into a clinically advantageous position away from the patient tocreate greater access, while avoiding arm collisions.

The cart base 15 balances the weight of the column 14, carriage 17, androbotic arms 12 over the floor. Accordingly, the cart base 15 housesheavier components, such as electronics, motors, power supply, as wellas components that either enable movement and/or immobilize the cart 11.For example, the cart base 15 includes rollable wheel-shaped casters 25that allow for the cart 11 to easily move around the room prior to aprocedure. After reaching the appropriate position, the casters 25 maybe immobilized using wheel locks to hold the cart 11 in place during theprocedure.

Positioned at the vertical end of the column 14, the console 16 allowsfor both a user interface for receiving user input and a display screen(or a dual-purpose device such as, for example, a touchscreen 26) toprovide the physician user with both preoperative and intraoperativedata. Potential preoperative data on the touchscreen 26 may includepreoperative plans, navigation and mapping data derived frompreoperative computerized tomography (CT) scans, and/or notes frompreoperative patient interviews. Intraoperative data on display mayinclude optical information provided from the tool, sensor andcoordinate information from sensors, as well as vital patientstatistics, such as respiration, heart rate, and/or pulse. The console16 may be positioned and tilted to allow a physician to access theconsole 16 from the side of the column 14 opposite the carriage 17. Fromthis position, the physician may view the console 16, robotic arms 12,and patient while operating the console 16 from behind the cart 11. Asshown, the console 16 also includes a handle 27 to assist withmaneuvering and stabilizing the cart 11.

FIG. 3 illustrates an embodiment of a robotically-enabled system 10arranged for ureteroscopy. In a ureteroscopy procedure, the cart 11 maybe positioned to deliver a ureteroscope 32, a procedure-specificendoscope designed to traverse a patient's urethra and ureter, to thelower abdominal area of the patient. In a ureteroscopy, it may bedesirable for the ureteroscope 32 to be directly aligned with thepatient's urethra to reduce friction and forces on the sensitive anatomyin the area. As shown, the cart 11 may be aligned at the foot of thetable to allow the robotic arms 12 to position the ureteroscope 32 fordirect linear access to the patient's urethra. From the foot of thetable, the robotic arms 12 may insert the ureteroscope 32 along thevirtual rail 33 directly into the patient's lower abdomen through theurethra.

After insertion into the urethra, using similar control techniques as inbronchoscopy, the ureteroscope 32 may be navigated into the bladder,ureters, and/or kidneys for diagnostic and/or therapeutic applications.For example, the ureteroscope 32 may be directed into the ureter andkidneys to break up kidney stone build up using a laser or ultrasoniclithotripsy device deployed down the working channel of the ureteroscope32. After lithotripsy is complete, the resulting stone fragments may beremoved using baskets deployed down the ureteroscope 32.

FIG. 4 illustrates an embodiment of a robotically-enabled system 10similarly arranged for a vascular procedure. In a vascular procedure,the system 10 may be configured such that the cart 11 may deliver amedical instrument 34, such as a steerable catheter, to an access pointin the femoral artery in the patient's leg. The femoral artery presentsboth a larger diameter for navigation as well as a relatively lesscircuitous and tortuous path to the patient's heart, which simplifiesnavigation. As in a ureteroscopy procedure, the cart 11 may bepositioned towards the patient's legs and lower abdomen to allow therobotic arms 12 to provide a virtual rail 35 with direct linear accessto the femoral artery access point in the patient's thigh/hip region.After insertion into the artery, the medical instrument 34 may bedirected and inserted by translating the instrument drivers 28.Alternatively, the cart 11 may be positioned around the patient's upperabdomen in order to reach alternative vascular access points, such as,for example, the carotid and brachial arteries near the shoulder andwrist.

B. Robotic System—Table.

Embodiments of the robotically-enabled medical system may alsoincorporate the patient's table. Incorporation of the table reduces theamount of capital equipment within the operating room by removing thecart, which allows greater access to the patient. FIG. 5 illustrates anembodiment of such a robotically-enabled system arranged for abronchoscopy procedure. System 36 includes a support structure or column37 for supporting platform 38 (shown as a “table” or “bed”) over thefloor. Much like in the cart-based systems, the end effectors of therobotic arms 39 of the system 36 comprise instrument drivers 42 that aredesigned to manipulate an elongated medical instrument, such as abronchoscope 40 in FIG. 5, through or along a virtual rail 41 formedfrom the linear alignment of the instrument drivers 42. In practice, aC-arm for providing fluoroscopic imaging may be positioned over thepatient's upper abdominal area by placing the emitter and detectoraround the table 38.

FIG. 6 provides an alternative view of the system 36 without the patientand medical instrument for discussion purposes. As shown, the column 37may include one or more carriages 43 shown as ring-shaped in the system36, from which the one or more robotic arms 39 may be based. Thecarriages 43 may translate along a vertical column interface 44 thatruns the length of the column 37 to provide different vantage pointsfrom which the robotic arms 39 may be positioned to reach the patient.The carriage(s) 43 may rotate around the column 37 using a mechanicalmotor positioned within the column 37 to allow the robotic arms 39 tohave access to multiples sides of the table 38, such as, for example,both sides of the patient. In embodiments with multiple carriages, thecarriages may be individually positioned on the column and may translateand/or rotate independently of the other carriages. While the carriages43 need not surround the column 37 or even be circular, the ring-shapeas shown facilitates rotation of the carriages 43 around the column 37while maintaining structural balance. Rotation and translation of thecarriages 43 allows the system 36 to align the medical instruments, suchas endoscopes and laparoscopes, into different access points on thepatient. In other embodiments (not shown), the system 36 can include apatient table or bed with adjustable arm supports in the form of bars orrails extending alongside it. One or more robotic arms 39 (e.g., via ashoulder with an elbow joint) can be attached to the adjustable armsupports, which can be vertically adjusted. By providing verticaladjustment, the robotic arms 39 are advantageously capable of beingstowed compactly beneath the patient table or bed, and subsequentlyraised during a procedure.

The robotic arms 39 may be mounted on the carriages 43 through a set ofarm mounts 45 comprising a series of joints that may individually rotateand/or telescopically extend to provide additional configurability tothe robotic arms 39. Additionally, the arm mounts 45 may be positionedon the carriages 43 such that, when the carriages 43 are appropriatelyrotated, the arm mounts 45 may be positioned on either the same side ofthe table 38 (as shown in FIG. 6), on opposite sides of the table 38 (asshown in FIG. 9), or on adjacent sides of the table 38 (not shown).

The column 37 structurally provides support for the table 38, and a pathfor vertical translation of the carriages 43. Internally, the column 37may be equipped with lead screws for guiding vertical translation of thecarriages 43, and motors to mechanize the translation of the carriages43 based the lead screws. The column 37 may also convey power andcontrol signals to the carriages 43 and the robotic arms 39 mountedthereon.

The table base 46 serves a similar function as the cart base 15 in thecart 11 shown in FIG. 2, housing heavier components to balance thetable/bed 38, the column 37, the carriages 43, and the robotic arms 39.The table base 46 may also incorporate rigid casters to providestability during procedures. Deployed from the bottom of the table base46, the casters may extend in opposite directions on both sides of thebase 46 and retract when the system 36 needs to be moved.

With continued reference to FIG. 6, the system 36 may also include atower (not shown) that divides the functionality of the system 36between the table and the tower to reduce the form factor and bulk ofthe table. As in earlier disclosed embodiments, the tower may be providea variety of support functionalities to the table, such as processing,computing, and control capabilities, power, fluidics, and/or optical andsensor processing. The tower may also be movable to be positioned awayfrom the patient to improve physician access and de-clutter theoperating room. Additionally, placing components in the tower allows formore storage space in the table base 46 for potential stowage of therobotic arms 39. The tower may also include a master controller orconsole that provides both a user interface for user input, such askeyboard and/or pendant, as well as a display screen (or touchscreen)for preoperative and intraoperative information, such as real-timeimaging, navigation, and tracking information. In some embodiments, thetower may also contain holders for gas tanks to be used forinsufflation.

In some embodiments, a table base may stow and store the robotic armswhen not in use. FIG. 7 illustrates a system 47 that stows robotic armsin an embodiment of the table-based system. In the system 47, carriages48 may be vertically translated into base 49 to stow robotic arms 50,arm mounts 51, and the carriages 48 within the base 49. Base covers 52may be translated and retracted open to deploy the carriages 48, armmounts 51, and robotic arms 50 around column 53, and closed to stow toprotect them when not in use. The base covers 52 may be sealed with amembrane 54 along the edges of its opening to prevent dirt and fluidingress when closed.

FIG. 8 illustrates an embodiment of a robotically-enabled table-basedsystem configured for a ureteroscopy procedure. In a ureteroscopy, thetable 38 may include a swivel portion 55 for positioning a patientoff-angle from the column 37 and table base 46. The swivel portion 55may rotate or pivot around a pivot point (e.g., located below thepatient's head) in order to position the bottom portion of the swivelportion 55 away from the column 37. For example, the pivoting of theswivel portion 55 allows a C-arm (not shown) to be positioned over thepatient's lower abdomen without competing for space with the column (notshown) below table 38. By rotating the carriage 35 (not shown) aroundthe column 37, the robotic arms 39 may directly insert a ureteroscope 56along a virtual rail 57 into the patient's groin area to reach theurethra. In a ureteroscopy, stirrups 58 may also be fixed to the swivelportion 55 of the table 38 to support the position of the patient's legsduring the procedure and allow clear access to the patient's groin area.

In a laparoscopy procedure, through small incision(s) in the patient'sabdominal wall, minimally invasive instruments may be inserted into thepatient's anatomy. In some embodiments, the minimally invasiveinstruments comprise an elongated rigid member, such as a shaft, whichis used to access anatomy within the patient. After inflation of thepatient's abdominal cavity, the instruments may be directed to performsurgical or medical tasks, such as grasping, cutting, ablating,suturing, etc. In some embodiments, the instruments can comprise ascope, such as a laparoscope. FIG. 9 illustrates an embodiment of arobotically-enabled table-based system configured for a laparoscopyprocedure. As shown, the carriages 43 of the system 36 may be rotatedand vertically adjusted to position pairs of the robotic arms 39 onopposite sides of the table 38, such that instrument 59 may bepositioned using the arm mounts 45 to be passed through minimalincisions on both sides of the patient to reach his/her abdominalcavity.

To accommodate laparoscopy procedures, the robotically-enabled tablesystem may also tilt the platform to a desired angle. FIG. 10illustrates an embodiment of the robotically-enabled medical system withpitch or tilt adjustment. As shown in FIG. 10, the system 36 mayaccommodate tilt of the table 38 to position one portion of the table ata greater distance from the floor than the other. Additionally, the armmounts 45 may rotate to match the tilt such that the robotic arms 39maintain the same planar relationship with the table 38. To accommodatesteeper angles, the column 37 may also include telescoping portions 60that allow vertical extension of the column 37 to keep the table 38 fromtouching the floor or colliding with the table base 46.

FIG. 11 provides a detailed illustration of the interface between thetable 38 and the column 37. Pitch rotation mechanism 61 may beconfigured to alter the pitch angle of the table 38 relative to thecolumn 37 in multiple degrees of freedom. The pitch rotation mechanism61 may be enabled by the positioning of orthogonal axes 1, 2 at thecolumn-table interface, each axis actuated by a separate motor 3, 4responsive to an electrical pitch angle command. Rotation along onescrew 5 would enable tilt adjustments in one axis 1, while rotationalong the other screw 6 would enable tilt adjustments along the otheraxis 2. In some embodiments, a ball joint can be used to alter the pitchangle of the table 38 relative to the column 37 in multiple degrees offreedom.

For example, pitch adjustments are particularly useful when trying toposition the table in a Trendelenburg position, i.e., position thepatient's lower abdomen at a higher position from the floor than thepatient's upper abdomen, for lower abdominal surgery. The Trendelenburgposition causes the patient's internal organs to slide towards his/herupper abdomen through the force of gravity, clearing out the abdominalcavity for minimally invasive tools to enter and perform lower abdominalsurgical or medical procedures, such as laparoscopic prostatectomy.

C. Instrument Driver & Interface.

The end effectors of the system's robotic arms may comprise (i) aninstrument driver (alternatively referred to as “instrument drivemechanism” or “instrument device manipulator”) that incorporateselectro-mechanical means for actuating the medical instrument and (ii) aremovable or detachable medical instrument which may be devoid of anyelectro-mechanical components, such as motors. This dichotomy may bedriven by the need to sterilize medical instruments used in medicalprocedures, and the inability to adequately sterilize expensive capitalequipment due to their intricate mechanical assemblies and sensitiveelectronics. Accordingly, the medical instruments may be designed to bedetached, removed, and interchanged from the instrument driver (and thusthe system) for individual sterilization or disposal by the physician orthe physician's staff. In contrast, the instrument drivers need not bechanged or sterilized, and may be draped for protection.

FIG. 12 illustrates an example instrument driver. Positioned at thedistal end of a robotic arm, instrument driver 62 comprises one or moredrive units 63 arranged with parallel axes to provide controlled torqueto a medical instrument via drive shafts 64. Each drive unit 63comprises an individual drive shaft 64 for interacting with theinstrument, a gear head 65 for converting the motor shaft rotation to adesired torque, a motor 66 for generating the drive torque, an encoder67 to measure the speed of the motor shaft and provide feedback to thecontrol circuitry, and control circuitry 68 for receiving controlsignals and actuating the drive unit. Each drive unit 63 may beindependently controlled and motorized, and the instrument driver 62 mayprovide multiple (e.g., four as shown in FIGS. 13 and 14) independentdrive outputs to the medical instrument. In operation, the controlcircuitry 68 would receive a control signal, transmit a motor signal tothe motor 66, compare the resulting motor speed as measured by theencoder 67 with the desired speed, and modulate the motor signal togenerate the desired torque.

For procedures that require a sterile environment, the robotic systemmay incorporate a drive interface, such as a sterile adapter connectedto a sterile drape, that sits between the instrument driver and themedical instrument. The chief purpose of the sterile adapter is totransfer angular motion from the drive shafts of the instrument driverto the drive inputs of the instrument while maintaining physicalseparation, and thus sterility, between the drive shafts and driveinputs. Accordingly, an example sterile adapter may comprise a series ofrotational inputs and outputs intended to be mated with the drive shaftsof the instrument driver and drive inputs on the instrument. Connectedto the sterile adapter, the sterile drape, comprised of a thin, flexiblematerial such as transparent or translucent plastic, is designed tocover the capital equipment, such as the instrument driver, robotic arm,and cart (in a cart-based system) or table (in a table-based system).Use of the drape would allow the capital equipment to be positionedproximate to the patient while still being located in an area notrequiring sterilization (i.e., non-sterile field). On the other side ofthe sterile drape, the medical instrument may interface with the patientin an area requiring sterilization (i.e., sterile field).

D. Medical Instrument.

FIG. 13 illustrates an example medical instrument with a pairedinstrument driver. Like other instruments designed for use with arobotic system, medical instrument 70 comprises an elongated shaft 71(or elongate body) and an instrument base 72. The instrument base 72,also referred to as an “instrument handle” due to its intended designfor manual interaction by the physician, may generally compriserotatable drive inputs 73, e.g., receptacles, pulleys, or spools, thatare designed to be mated with drive outputs 74 that extend through adrive interface on instrument driver 75 at the distal end of robotic arm76. When physically connected, latched, and/or coupled, the mated driveinputs 73 of the instrument base 72 may share axes of rotation with thedrive outputs 74 in the instrument driver 75 to allow the transfer oftorque from the drive outputs 74 to the drive inputs 73. In someembodiments, the drive outputs 74 may comprise splines that are designedto mate with receptacles on the drive inputs 73.

The elongated shaft 71 is designed to be delivered through either ananatomical opening or lumen, e.g., as in endoscopy, or a minimallyinvasive incision, e.g., as in laparoscopy. The elongated shaft 71 maybe either flexible (e.g., having properties similar to an endoscope) orrigid (e.g., having properties similar to a laparoscope) or contain acustomized combination of both flexible and rigid portions. Whendesigned for laparoscopy, the distal end of a rigid elongated shaft maybe connected to an end effector extending from a jointed wrist formedfrom a clevis with at least one degree of freedom and a surgical tool ormedical instrument, such as, for example, a grasper or scissors, thatmay be actuated based on the force from the tendons as the drive inputsrotate in response to torque received from the drive outputs 74 of theinstrument driver 75. When designed for endoscopy, the distal end of aflexible elongated shaft may include a steerable or controllable bendingsection that may be articulated and bent based on the torque receivedfrom the drive outputs 74 of the instrument driver 75.

Torque from the instrument driver 75 is transmitted down the elongatedshaft 71 using tendons along the elongated shaft 71. These individualtendons, such as pull wires, may be individually anchored to individualdrive inputs 73 within the instrument handle 72. From the handle 72, thetendons are directed down one or more pull lumens along the elongatedshaft 71 and anchored at the distal portion of the elongated shaft 71 orin the wrist at the distal portion of the elongated shaft 71. During asurgical procedure, such as a laparoscopic, endoscopic or hybridprocedure, these tendons may be coupled to a distally mounted endeffector, such as a wrist, grasper, or scissor. Under such anarrangement, torque exerted on the drive inputs 73 would transfertension to the tendon, thereby causing the end effector to actuate insome way. In laparoscopy, the tendon may cause a joint to rotate aboutan axis, thereby causing the end effector to move in one direction oranother. Alternatively, the tendon may be connected to one or more jawsof a grasper at the distal end of the elongated shaft 71, where thetension from the tendon causes the grasper to close.

In endoscopy, the tendons may be coupled to a bending or articulatingsection positioned along the elongated shaft 71 (e.g., at the distalend) via adhesive, control ring, or other mechanical fixation. Whenfixedly attached to the distal end of a bending section, torque exertedon the drive inputs 73 would be transmitted down the tendons, causingthe softer, bending section (sometimes referred to as the articulablesection or region) to bend or articulate. Along the non-bendingsections, it may be advantageous to spiral or helix the individual pulllumens that direct the individual tendons along (or inside) the walls ofthe endoscope shaft to balance the radial forces that result from thetension in the pull wires. The angle of the spiraling and/or spacingtherebetween may be altered or engineered for specific purposes, whereintighter spiraling exhibits lesser shaft compression under load forces,while lower amounts of spiraling results in greater shaft compressionunder load forces, but also exhibits more limited bending. On the otherend of the spectrum, the pull lumens may be directed parallel to thelongitudinal axis of the elongated shaft 71 to allow for controlledarticulation in the desired bending or articulable sections.

In endoscopy, the elongated shaft 71 houses a number of components toassist with the robotic procedure. The shaft 71 may comprise a workingchannel for deploying surgical tools (or medical instruments),irrigation, and/or aspiration to the operative region at the distal endof the shaft 71. The shaft 71 may also accommodate wires and/or opticalfibers to transfer signals to/from an optical assembly at the distaltip, which may include an optical camera. The shaft 71 may alsoaccommodate optical fibers to carry light from proximally-located lightsources, such as light emitting diodes, to the distal end of the shaft71.

At the distal end of the instrument 70, the distal tip may also comprisethe opening of a working channel for delivering tools for diagnosticand/or therapy, irrigation, and aspiration to an operative site. Thedistal tip may also include a port for a camera, such as a fiberscope ora digital camera, to capture images of an internal anatomical space.Relatedly, the distal tip may also include ports for light sources forilluminating the anatomical space when using the camera.

In the example of FIG. 13, the drive shaft axes, and thus the driveinput axes, are orthogonal to the axis of the elongated shaft 71. Thisarrangement, however, complicates roll capabilities for the elongatedshaft 71. Rolling the elongated shaft 71 along its axis while keepingthe drive inputs 73 static results in undesirable tangling of thetendons as they extend off the drive inputs 73 and enter pull lumenswithin the elongated shaft 71. The resulting entanglement of suchtendons may disrupt any control algorithms intended to predict movementof the flexible elongated shaft 71 during an endoscopy procedure.

FIG. 14 illustrates an alternative design for an instrument driver andinstrument where the axes of the drive units are parallel to the axis ofthe elongated shaft of the instrument. As shown, a circular instrumentdriver 80 comprises four drive units with their drive outputs 81 alignedin parallel at the end of a robotic arm 82. The drive units, and theirrespective drive outputs 81, are housed in a rotational assembly 83 ofthe instrument driver 80 that is driven by one of the drive units withinthe assembly 83. In response to torque provided by the rotational driveunit, the rotational assembly 83 rotates along a circular bearing thatconnects the rotational assembly 83 to the non-rotational portion 84 ofthe instrument driver 80. Power and controls signals may be communicatedfrom the non-rotational portion 84 of the instrument driver 80 to therotational assembly 83 through electrical contacts that may bemaintained through rotation by a brushed slip ring connection (notshown). In other embodiments, the rotational assembly 83 may beresponsive to a separate drive unit that is integrated into thenon-rotatable portion 84, and thus not in parallel to the other driveunits. The rotational mechanism 83 allows the instrument driver 80 torotate the drive units, and their respective drive outputs 81, as asingle unit around an instrument driver axis 85.

Like earlier disclosed embodiments, an instrument 86 may comprise anelongated shaft portion 88 and an instrument base 87 (shown with atransparent external skin for discussion purposes) comprising aplurality of drive inputs 89 (such as receptacles, pulleys, and spools)that are configured to receive the drive outputs 81 in the instrumentdriver 80. Unlike prior disclosed embodiments, the instrument shaft 88extends from the center of the instrument base 87 with an axissubstantially parallel to the axes of the drive inputs 89, rather thanorthogonal as in the design of FIG. 13.

When coupled to the rotational assembly 83 of the instrument driver 80,the medical instrument 86, comprising instrument base 87 and instrumentshaft 88, rotates in combination with the rotational assembly 83 aboutthe instrument driver axis 85. Since the instrument shaft 88 ispositioned at the center of instrument base 87, the instrument shaft 88is coaxial with instrument driver axis 85 when attached. Thus, rotationof the rotational assembly 83 causes the instrument shaft 88 to rotateabout its own longitudinal axis. Moreover, as the instrument base 87rotates with the instrument shaft 88, any tendons connected to the driveinputs 89 in the instrument base 87 are not tangled during rotation.Accordingly, the parallelism of the axes of the drive outputs 81, driveinputs 89, and instrument shaft 88 allows for the shaft rotation withouttangling any control tendons.

E. Navigation and Control.

Traditional endoscopy may involve the use of fluoroscopy (e.g., as maybe delivered through a C-arm) and other forms of radiation-based imagingmodalities to provide endoluminal guidance to an operator physician. Incontrast, the robotic systems contemplated by this disclosure canprovide for non-radiation-based navigational and localization means toreduce physician exposure to radiation and reduce the amount ofequipment within the operating room. As used herein, the term“localization” may refer to determining and/or monitoring the positionof objects in a reference coordinate system. Technologies such aspreoperative mapping, computer vision, real-time EM tracking, and robotcommand data may be used individually or in combination to achieve aradiation-free operating environment. In other cases, whereradiation-based imaging modalities are still used, the preoperativemapping, computer vision, real-time EM tracking, and robot command datamay be used individually or in combination to improve upon theinformation obtained solely through radiation-based imaging modalities.

FIG. 15 is a block diagram illustrating a localization system 90 thatestimates a location of one or more elements of the robotic system, suchas the location of the instrument, in accordance to an exampleembodiment. The localization system 90 may be a set of one or morecomputer devices configured to execute one or more instructions. Thecomputer devices may be embodied by a processor (or processors) andcomputer-readable memory in one or more components discussed above. Byway of example and not limitation, the computer devices may be in thetower 30 shown in FIG. 1, the cart 11 shown in FIGS. 1-4, the beds shownin FIGS. 5-10, etc.

As shown in FIG. 15, the localization system 90 may include alocalization module 95 that processes input data 91-94 to generatelocation data 96 for the distal tip of a medical instrument. Thelocation data 96 may be data or logic that represents a location and/ororientation of the distal end of the instrument relative to a frame ofreference. The frame of reference can be a frame of reference relativeto the anatomy of the patient or to a known object, such as an EM fieldgenerator (see discussion below for the EM field generator).

The various input data 91-94 are now described in greater detail.Preoperative mapping may be accomplished through the use of thecollection of low dose computed tomography (CT) scans. Preoperative CTscans are reconstructed into three-dimensional images, which arevisualized, e.g. as “slices” of a cutaway view of the patient's internalanatomy. When analyzed in the aggregate, image-based models foranatomical cavities, spaces, and structures of the patient's anatomy,such as a patient lung network, may be generated. Techniques such ascenter-line geometry may be determined and approximated from the CTimages to develop a three-dimensional volume of the patient's anatomy,referred to model data 91 (also referred to as “preoperative model data”when generated using only preoperative CT scans). In some embodiments,the preoperative model data 91 may include data from, e.g., fluoroscopy,magnetic resonance imaging (MRI), ultrasound imaging, and/or x-rays. Theuse of center-line geometry is discussed in U.S. patent application Ser.No. 14/523,760, the contents of which are herein incorporated in itsentirety. Network topological models may also be derived from the CTimages, and are particularly appropriate for bronchoscopy.

In some embodiments, the instrument may be equipped with a camera toprovide vision data (or image data) 92. The localization module 95 mayprocess the vision data 92 to enable one or more vision-based (orimage-based) location tracking modules or features. For example, thepreoperative model data 91 may be used in conjunction with the visiondata 92 to enable computer vision-based tracking of the medicalinstrument (e.g., an endoscope or an instrument advance through aworking channel of the endoscope). For example, using the preoperativemodel data 91, the robotic system may generate a library of expectedendoscopic images from the model based on the expected path of travel ofthe endoscope, each image linked to a location within the model.Intraoperatively, this library may be referenced by the robotic systemin order to compare real-time images captured at the camera (e.g., acamera at a distal end of the endoscope) to those in the image libraryto assist localization.

Other computer vision-based tracking techniques use feature tracking todetermine motion of the camera, and thus the endoscope. Some features ofthe localization module 95 may identify circular geometries in thepreoperative model data 91 that correspond to anatomical lumens andtrack the change of those geometries to determine which anatomical lumenwas selected, as well as the relative rotational and/or translationalmotion of the camera. Use of a topological map may further enhancevision-based algorithms or techniques.

Optical flow, another computer vision-based technique, may analyze thedisplacement and translation of image pixels in a video sequence in thevision data 92 to infer camera movement. Examples of optical flowtechniques may include motion detection, object segmentationcalculations, luminance, motion compensated encoding, stereo disparitymeasurement, etc. Through the comparison of multiple frames overmultiple iterations, movement and location of the camera (and thus theendoscope) may be determined.

The localization module 95 may use real-time EM tracking to generate areal-time location of the endoscope in a global coordinate system thatmay be registered to the patient's anatomy, represented by thepreoperative model. In EM tracking, an EM sensor (or tracker) comprisingone or more sensor coils embedded in one or more locations andorientations in a medical instrument (e.g., an endoscopic tool) measuresthe variation in the EM field created by one or more static EM fieldgenerators positioned at a known location. The location informationdetected by the EM sensors is stored as EM data 93. The EM fieldgenerator (or transmitter), may be placed close to the patient to createa low intensity magnetic field that the embedded sensor may detect. Themagnetic field induces small currents in the sensor coils of the EMsensor, which may be analyzed to determine the distance and anglebetween the EM sensor and the EM field generator. These distances andorientations may be intraoperatively “registered” to the patient anatomy(e.g., the preoperative model) in order to determine the geometrictransformation that aligns a single location in the coordinate systemwith a position in the preoperative model of the patient's anatomy. Onceregistered, an embedded EM tracker in one or more positions of themedical instrument (e.g., the distal tip of an endoscope) may providereal-time indications of the progression of the medical instrumentthrough the patient's anatomy.

Robotic command and kinematics data 94 may also be used by thelocalization module 95 to provide localization data 96 for the roboticsystem. Device pitch and yaw resulting from articulation commands may bedetermined during preoperative calibration. Intraoperatively, thesecalibration measurements may be used in combination with known insertiondepth information to estimate the position of the instrument.Alternatively, these calculations may be analyzed in combination withEM, vision, and/or topological modeling to estimate the position of themedical instrument within the network.

As FIG. 15 shows, a number of other input data can be used by thelocalization module 95. For example, although not shown in FIG. 15, aninstrument utilizing shape-sensing fiber can provide shape data that thelocalization module 95 can use to determine the location and shape ofthe instrument.

The localization module 95 may use the input data 91-94 incombination(s). In some cases, such a combination may use aprobabilistic approach where the localization module 95 assigns aconfidence weight to the location determined from each of the input data91-94. Thus, where the EM data may not be reliable (as may be the casewhere there is EM interference) the confidence of the locationdetermined by the EM data 93 can be decrease and the localization module95 may rely more heavily on the vision data 92 and/or the roboticcommand and kinematics data 94.

As discussed above, the robotic systems discussed herein may be designedto incorporate a combination of one or more of the technologies above.The robotic system's computer-based control system, based in the tower,bed, and/or cart, may store computer program instructions, for example,within a non-transitory computer-readable storage medium such as apersistent magnetic storage drive, solid state drive, or the like, that,upon execution, cause the system to receive and analyze sensor data anduser commands, generate control signals throughout the system, anddisplay the navigational and localization data, such as the position ofthe instrument within the global coordinate system, anatomical map, etc.

II. Navigation of Luminal Networks

The various robotic systems discussed above can be employed to perform avariety of medical procedures, such as endoscopic and laparoscopyprocedures. During certain procedures, a medical instrument, such as arobotically-controlled medical instrument, is inserted into a patient'sbody. Within the patient's body, the instrument may be positioned withina luminal network of the patient. As used herein, the term “luminalnetwork” refers to any cavity structure within the body, whethercomprising a plurality of lumens or branches (e.g., a plurality ofbranched lumens, as in the lung or blood vessels) or a single lumen orbranch (e.g., within the gastrointestinal tract). During the procedure,the instrument may be moved (e.g., navigated, guided, driven, etc.)through the luminal network to one or more areas of interest. Movementof the instrument through the system may be aided by the navigation orlocalization system 90 discussed above, which can provide positionalinformation about the instrument to a physician controlling the roboticsystem.

FIG. 16 illustrates an example luminal network 130 of a patient. In theillustrated embodiment, the luminal network 130 is a bronchial networkof airways 150 (i.e., lumens, branches) of the patient's lung. Althoughthe illustrated luminal network 130 is a bronchial network of airwayswithin the patient's lung, this disclosure is not limited to only theillustrated example. The robotic systems and methods described hereinmay be used to navigate any type of luminal network, such as bronchialnetworks, renal networks, cardiovascular networks (e.g., arteries andveins), gastrointestinal tracts, urinary tracts, etc.

As illustrated, the luminal network 130 comprises a plurality of airways150 that are arranged in a branched structure. In general, the luminalnetwork 130 comprises a three-dimensional structure. For ease ofillustration, FIG. 16 represents the luminal network 130 as atwo-dimensional structure. This should not be construed to limit thepresent disclosure to two-dimensional luminal networks in any way.

FIG. 16 also illustrates an example of a medical instrument positionedwithin the luminal network 130. The medical instrument is navigatedthrough the luminal network 130 towards an area of interest (e.g.,nodule 155) for diagnosis and/or treatment. In the illustrated example,the nodule 155 is located at the periphery of the airways 150, althoughthe area(s) of interest can be positioned anywhere within the luminalnetwork 130 depending on the patient and procedure.

In the illustrated example, the medical instrument comprises anendoscope 115. The endoscope 115 can include a sheath 120 and a leader145. In some embodiments, the sheath 120 and the leader 145 may bearranged in a telescopic manner. For example, the leader 145 may beslidably positioned inside a working channel of the sheath 120. Thesheath 120 may have a first diameter, and its distal end may not be ableto be positioned through the smaller-diameter airways 150 around thenodule 155. Accordingly, the leader 145 may be configured to extend fromthe working channel of the sheath 120 the remaining distance to thenodule 155. The leader 145 may have a lumen through which instruments,for example biopsy needles, cytology brushes, and/or tissue samplingforceps, can be passed to the target tissue site of the nodule 155. Insuch implementations, both the distal end of the sheath 120 and thedistal end of the leader 145 can be provided with EM instrument sensors(e.g., EM instrument sensors 305 in FIG. 18) for tracking their positionwithin the airways 150. This telescopic arrangement of the sheath 120and the leader 145 may allow for a thinner design of the endoscope 115and may improve a bend radius of the endoscope 115 while providing astructural support via the sheath 120.

In other embodiments, the overall diameter of the endoscope 115 may besmall enough to reach the periphery without the telescopic arrangement,or may be small enough to get close to the periphery (e.g., within about2.5-3 cm) to deploy medical instruments through a non-steerablecatheter. The medical instruments deployed through the endoscope 115 maybe equipped with EM instrument sensors (e.g., EM instrument sensors 305in FIG. 18), and the image-based airway analysis and mapping techniquesdescribed below can be applied to such medical instruments.

As shown, to reach the nodule 155, the instrument (e.g., endoscope 115)must be navigated or guided through the airways 150 of the luminalnetwork. An operator (such as a physician) can control the roboticsystem to navigate the instrument to the nodule 155. The operator mayprovide inputs for controlling the robotic system.

FIG. 17 illustrates an example command console 200 that can be used withsome implementations of the robotic systems described herein. Theoperator may provide the inputs for controlling the robotic system, forexample, to navigate or guide the instrument to an area of interest suchas nodule 155, via the command console 200. The command console 200 maybe embodied in a wide variety of arrangements or configurations. In theillustrated example, the command console 200 includes a console base201, displays 202 (e.g., monitors), and one or more control modules(e.g., keyboard 203 and joystick 204). A user 205 (e.g., physician orother operator) can remotely control the medical robotic system (e.g.,the systems described with reference to FIGS. 1-15) from an ergonomicposition using the command console 200.

The displays 202 may include electronic monitors (e.g., LCD displays,LED displays, touch-sensitive displays), virtual reality viewing devices(e.g., goggles or glasses), and/or other display devices. In someembodiments, one or more of the displays 202 displays positioninformation about the instrument, for example, as determined by thelocalization system 90 (FIG. 15). In some embodiments, one or more ofthe displays 202 displays a preoperative model of the patient's luminalnetwork 130. The positional information can be overlaid on thepreoperative model. The displays 202 can also display image informationreceived from a camera or another sensing device positioned on theinstrument within the luminal network 130. In some embodiments, a modelor representation of the instrument is displayed with the preoperativemodel to help indicate a status of a surgical or medical procedure.

In some embodiments, the console base 201 includes a central processingunit (e.g., CPU or processor), a memory unit (e.g., computer-readablememory), a data bus, and associated data communication ports that areresponsible for interpreting and processing signals such as cameraimagery and tracking sensor data, e.g., from a medical instrumentpositioned within a luminal network of a patient.

The console base 201 may also process commands and instructions providedby the user 205 through control modules 203, 204. In addition to thekeyboard 203 and joystick 204 shown in FIG. 17, the control modules mayinclude other devices, such as computer mice, trackpads, trackballs,control pads, controllers such as handheld remote controllers, andsensors (e.g., motion sensors or cameras) that capture hand gestures andfinger gestures. A controller can include a set of user inputs (e.g.,buttons, joysticks, directional pads, etc.) mapped to an operation ofthe instrument (e.g., articulation, driving, water irrigation, etc.).Using the control modules 203, 204 of the command console 200, the user205 may navigate an instrument through the luminal network 130.

FIG. 18 illustrates a detail view of a distal end of an example medicalinstrument 300. The medical instrument 300 may be representative of theendoscope 115 or steerable catheter 145 of FIG. 16. The medicalinstrument 300 may be representative of any medical instrument describedthroughout the disclosure, such as the endoscope 13 of FIG. 1, theureteroscope 32 of FIG. 3, the laparoscope 59 of FIG. 9, etc. In FIG.18, the distal end of the instrument 300 includes an imaging device 315,illumination sources 310, and ends of EM sensor coils 305, which form anEM instrument sensor. The distal end further includes an opening to aworking channel 320 of the instrument 300 through which surgicalinstruments, such as biopsy needles, cytology brushes, forceps, etc.,may be inserted along the instrument shaft, allowing access to the areanear the instrument tip.

EM coils 305 located on the distal end of the instrument 300 may be usedwith an EM tracking system to detect the position and orientation of thedistal end of the instrument 300 while it is positioned within a luminalnetwork. In some embodiments, the coils 305 may be angled to providesensitivity to EM fields along different axes, giving the disclosednavigational systems the ability to measure a full 6 degrees of freedom(DoF): three positional DoF and three angular DoF. In other embodiments,only a single coil 305 may be disposed on or within the distal end withits axis oriented along the instrument shaft. Due to the rotationalsymmetry of such a system, it may be insensitive to roll about its axis,so only five degrees of freedom may be detected in such animplementation. Alternatively or additionally, other types of positionsensors may be employed.

The illumination sources 310 provide light to illuminate a portion of ananatomical space. The illumination sources 310 can each be one or morelight-emitting devices configured to emit light at a selected wavelengthor range of wavelengths. The wavelengths can be any suitable wavelength,for example, visible spectrum light, infrared light, x-ray (e.g., forfluoroscopy), to name a few examples. In some embodiments, theillumination sources 310 can include light-emitting diodes (LEDs)located at the distal end of the instrument 300. In some embodiments,the illumination sources 310 can include one or more fiber optic fibersextending through the length of the endoscope to transmit light throughthe distal end from a remote light source, for example, an x-raygenerator. Where the distal end includes multiple illumination sources310, these can each be configured to emit the same or differentwavelengths of light as one another.

The imaging device 315 can include any photosensitive substrate orstructure configured to convert energy representing received light intoelectric signals, for example, a charge-coupled device (CCD) orcomplementary metal-oxide semiconductor (CMOS) image sensor. Someexamples of the imaging device 315 can include one or more opticalfibers, for example, a fiber optic bundle, configured to transmit lightrepresenting an image from the distal end 300 of the endoscope to aneyepiece and/or image sensor near the proximal end of the endoscope. Theimaging device 315 can additionally include one or more lenses and/orwavelength pass or cutoff filters as required for various opticaldesigns. The light emitted from the illumination sources 310 allows theimaging device 315 to capture images of the interior of a patient'sluminal network. These images can then be transmitted as individualframes or series of successive frames (e.g., a video) to a computersystem such as command console 200. As mentioned above and as will bedescribed in greater detail below, the images captured by the imagingdevice 315 (e.g., vision data 92 of FIG. 15) can be utilized by thenavigation or localization system 95 to determine or estimate theposition of the instrument (e.g., the position of the distal tip of theinstrument 300) within a luminal network.

III. Image-Based Airway Analysis and Mapping

Embodiments of the disclosure relate to systems and techniques forimage-based airway analysis and mapping. As used herein, image-basedairway analysis may refer to identifying within an image one or moreairways associated with one or more branches of a luminal network anddetermining branching information indicative of how the current airwayin which the image is captured branches into the detected “child”airways. For example, an image-based airway analysis system may capturean image of an interior of a luminal network using an imaging devicepositioned on an instrument within the luminal network, and theimage-based airway analysis system may analyze the image to identify oneor more airways shown in the image. As used herein, image-based airwaymapping may refer to mapping the airways identified in the image tocorresponding airways or branches of the luminal network indicated by,for example, the preoperative model data. For example, an image-basedairway mapping system may be configured to identify which airways orbranches of a given luminal network correspond to the airways orbranches identified in the captured images. These systems and techniquesmay be used to determine or estimate the position of an instrumentwithin the luminal network. In certain implementations, these systemsand techniques may be used in conjunction with various other navigationand localization modalities (e.g., as described above with reference toFIG. 15).

A. Overview of Image-Based Airway Analysis and Mapping.

The ability to navigate inside a luminal network may be a feature of therobotically-controlled surgical systems described herein. As usedherein, navigation may refer to locating or determining the position ofan instrument within a luminal network. The determined position may beused to help guide the instrument to one or more particular areas ofinterest within the luminal network. In some embodiments, therobotically-controlled surgical systems utilize one or more independentsensing modalities to provide intraoperative navigation for theinstrument. As shown in FIG. 15, the independent sensing modalities mayprovide position data (e.g., EM data 93), vision data 92, and/or roboticcommand and kinematics data 94. These independent sensing modalities mayinclude estimation modules configured to provide independent estimatesof position. The independent estimates can then be combined into onenavigation output, for example, using localization module 95, which canbe used by the system or displayed to the user. Image-based airwayanalysis and mapping may provide an independent sensing modality, basedon vision data 92, that can provide an independent estimate of position.In particular, in some instances image-based airway analysis and mappingprovides a combination of a sensing modality and a state/positionestimation module that estimates which lumen or branch of a luminalnetwork an imaging device of the instrument is located based on one ormore images captured by the imaging device. In some embodiments, theestimate provided by image-based airway analysis and mapping may be usedalone or with other position estimates to determine a final positionestimate that can be used by the system or displayed to the user. Insome embodiments, the image-based airway analysis and mapping systemsand methods described herein may base its estimate on prior positionestimates determined using a plurality sensing modalities.

FIG. 19 illustrates an example method 400 for image-based airwayanalysis and mapping. The method 400 may be implemented in variousrobotically-controlled surgical systems as described herein. The method400 may include two steps or blocks: detecting airways in an image(block 402) and mapping the detected airways to corresponding airways(e.g., indicated by the preoperative model data) of the luminal network(block 404).

At block 402, the method 400 detects one or more airways within animage. As noted above, during a medical procedure, an instrument may bepositioned within a luminal network (see FIG. 16). As shown in FIG. 18,the instrument may include an imaging device 315 (such as a camera)positioned thereon. The imaging device 315 may capture images of theinterior of the luminal network. For example, at a particular instant,the imaging device 315 may capture an image of the interior of theparticular airway of the luminal network in which the instrument iscurrently positioned. At block 402, the method 400 can analyze the imageto detect one or more airways within the image. For example, an airwaydetected within the image may be a set of pixels that satisfy an airwaydetection condition (e.g., having a pixel intensity greater than athreshold value). The detected airway may be the current airway in whichthe image is captured by the imaging device 315 and/or one or more childairways (e.g., subsequent branches of the luminal network) of thecurrent airway. Block 402 may involve image analysis that processes animage to determine the number and location of the airways shown in theimage. In certain implementations, if the image is determined to containone or more airways, various features of the airways may also bedetermined. Such features may include determining a shape or contour ofthe detected airways and/or determining a centroid of the detectedairways.

At block 404, the method 400 maps the one or more detected airways tospecific airways of the luminal network. At block 404, the method 400determines which airways of the luminal network branches into whichother subsequent airways of the luminal network. For example, based oncertain preoperative model data (e.g., CT data), the system may be awareof the airways that are expected to be captured in a given image (e.g.,based on the current location of the imaging device 315). Upon detectingone or more airways in the image at block 402, the method 400 may mapthe detected airways to the expected airways based on the preoperativemodel data.

By mapping the airways detected in the camera image to correspondingexpected airways in the luminal network, the method 400 may provide anestimate of position for the instrument. For example, using the method400, the system or the instrument can identify which airways theinstrument “sees” and use this information to estimate, within theluminal network, the airway in which the instrument is currently locatedand the airway that the instrument is about to enter.

B. Image-Based Airway Analysis.

Image-based airway analysis may include analyzing an image captured bythe imaging device 315 of an instrument positioned within a luminalnetwork to detect one or more airways in the image. FIG. 20 provides anexample image 500 of an interior of a branch of a luminal network. Inthe illustrated example, the image 500 is an interior image of an airwayof a lung, although the image 500 may be representative of any type ofluminal network. Two subsequent airways 502 are shown in the image 500.The airways 502 are connected to the current airway from which the imageis captured.

Image-based airway analysis can include a method whereby a computersystem can recognize the airways 502 computationally. In some cases, theimage 500 includes two classes of pixels: (1) pixels representing wallsof the luminal network (e.g., tissue), and (2) pixels representingairways (e.g., airway openings). According to certain embodiments, theimage-based airway analysis can systematically detect these two classesof pixels to identify and detect airways within an image. For example,the airways 502 may be detected by classifying the pixels into these twoclasses based on the pixel intensity. Image analysis and detectionmethods based on pixel intensity are described in greater detail in U.S.patent application Ser. No. 15/783,903, the contents of which are hereinincorporated in its entirety.

C. Image-Based Airway Mapping

Image-based airway mapping techniques described herein may be used todetermine which branches of the luminal network are associated with thedetected branches or airways. That is, image-based airway mapping candetermine which subsequent branches of the luminal network are connectedto the current branch from which the image is captured. By mapping thedetected airways to branches of the luminal network, the position of theinstrument within the luminal network can be determined. Further, anestimate or prediction of which branch the instrument will be moved intocan also be obtained.

In some embodiments, detected airways can be mapped to branches of theluminal network by comparing features of the detected airways tofeatures of the branches of the luminal network. The features of thedetected airways may be determined through image analysis as describedabove. The features of the branches of the luminal network can bedetermined from a model of the luminal network, such as a preoperativemodel of the luminal network. Further, in certain embodiments, mappingdetected airways to branches of the luminal network can be based on acurrent position estimate of the instrument within the luminal network.The current position estimate can be determined based on various sensingmodalities as described above with reference to FIG. 15. Mappingdetected airways to branches of the luminal network based on a currentposition estimate can improve the efficiency, speed, and/or accuracy ofthe mapping process. For example, given a current position estimate,features of the detected airways can be compared to features of expectedsubsequent branches. This may minimize the computational load requiredto perform the mapping and improve mapping speed.

FIG. 21 illustrates a simplified representation of a luminal network1000. The luminal network 1000 comprises a plurality of branches (e.g.,lumens, segments, airways, etc.) 1002, 1004, 1006, 1008, 1010, 1012,1014, 1016. The luminal network 1000 also comprises bifurcations 1020,1022 and trifurcation 1024 connecting various branches to each other. Inthe example of FIG. 21, the branches 1004, 1006 are in the samegeneration, and the branches 1008, 1010, 1012, 1014, 1016 are in thesame generation. In some embodiments, the luminal network 1000 mayrepresent a portion of a bronchial network of a lung, and the branchesmay represent airways. Although many example embodiments of the presentdisclosure are described with reference to airways in a bronchialnetwork, the embodiments are not limited as such and may be extended toother types of branches in other types of luminal networks.

The luminal network 1000 may be represented by a model. In someembodiments, the model is determined preoperatively. For example,preoperative model data 91 (i.e., information about the preoperativemodel) may be stored and made available to the navigation andlocalization system 90 (FIG. 15). In other embodiments, at least aportion of the model is determined intraoperatively. As will bedescribed below as an example, image-based airway mapping can beconfigured to map the detected branches to the branches of the luminalnetwork 1000 generated based on the preoperative model data 91.

D. Potential Challenges with Image-Based Airway Detection

In some cases, the images captured by a bronchoscopic camera may containairways from different generations. In such cases, the number of airwaysdetected by the image-based airway detection methods described hereinmay be incorrect. For example, FIG. 22 shows image 2200(a) (left) andimage 2200(b) (right) captured in the branch 1002 of the luminal network1000 of FIG. 21. In the example of FIG. 22, airway projections have beenadded to the images 2200(a), 2200(b) to indicate the number andlocations of the branches expected to be seen from the current cameralocation.

The image-based airway detection method performed on the image 2200(a)has resulted in four detected airways as shown in the image 2200(a).Such a result may cause the system to determine that the current airwaybranches into the four detected branches. However, as indicated by thehierarchical structure of FIG. 21, such a determination would beincorrect. The preoperative model data (e.g., hierarchical structure ofthe luminal network 1000 illustrated in FIG. 21) indicates (correctly,and in a manner consistent with the preoperative model data) that thebranch 1002 from which the image 2200(a) is captured is expected tobranch into two branches (i.e. branches 1004 and 1006), and not fourbranches as detected in the image 2200(a). While the image 2200(a)clearly shows four airways, three of the airways pictured at the bottomright corner of the image 2200(a) are “grandchildren” (i.e. branches1012, 1014, 1016) of the current branch 1002 that branched out from thebranch 1006 that is immediately subsequent to the current branch 1002(i.e. “child” of the current branch 1002).

Thus, in some embodiments, by utilizing the preoperative model dataindicative of the number (i.e. count and/or location) of airways to bepresent in a given image, the airway analysis and mapping systems andmethods of the present disclosure can detect the correct number ofairways, as illustrated in the image 2200(b).

E. Airway Analysis

FIG. 23A illustrates an example airway analysis and mapping system2300A. As shown, the airway analysis and mapping system 2300A includesan airway tracking module 2300A-1, an airway association module 2300A-2,and an airway merging module 2300A-3. The airway analysis and mappingsystem 2300A receives as input (i) image data (e.g., image capturedusing an imaging device) from an image data store 2310 (ii) model data(e.g., preoperative model data such as fluoroscopy data, MRI data,ultrasound imaging data, x-ray data, and the like) from a preoperativemodel data store 2320, (iii) sensor data (e.g., EM data, vision data,robot data, shape sensing fiber data, and the like) from a sensor datastore 2330 and outputs airway data (e.g., location, size, and center ofidentified or merged airways, mapping between identified or mergedairways and preoperative model data, and the like) to the airway datastore 2340. The airway tracking module 2300A-1 identifies the airway(s)in each image in the image data. The airway association module 2300A-2associates the airway(s)s in current image with the airway(s) in priorimages. The airway merging module 2300A-3 determines whether two or moreairways should be merged based on their relationship, for example, usingthe sensor data and the preoperative model data. The process ofidentifying, associating, and merging airways according to variousembodiments is described in greater detail below with reference to FIGS.23B, 24, and 25.

FIG. 23B illustrates the temporal context in which the image-basedairway analysis according to example embodiments is performed. Forsimplicity, one or more steps are described as being performed by asystem (e.g., the airway analysis and mapping system 2300A).

As shown, a series of images (i.e., images t₀, t₁, . . . ) are capturedby an imaging device on a medical instrument navigating the luminalnetwork of a patient, and images captured at different times (e.g., t₀and t₁) undergo airway tracking, airway association, and airway merging.At a high level, this process of performing airway tracking, airwayassociation, and airway merging analyzes a stream of images captured bythe imaging device to compare locations of currently detected airwayswith estimated locations of airways detected in prior images and thendetermines a relationship between the airways detected in the currentand prior images to determine whether the airways should be merged. Thestream of images, or data derived therefrom, such as airway locationinformation, may be stored by the system to perform the analysis.Temporal airway tracking, airway association, and airway merging are nowdescribed in greater detail.

FIG. 24 illustrates the spatial nature of the image-based airwayanalysis according to example embodiments. As shown, the airways in theprevious and current images can exhibit a variety of different types ofspatial relationship (e.g., one-to-one, one-to-many, many-to-one,zero-to-one, and one-to-zero). These examples are described in greaterdetail below in connection with Airway Association.

E.1. Temporal Airway Tracking

At a high level, image-based temporal tracking involves estimating thestatus (e.g., locations and sizes) of the airways in a current imagerelative to the status of the airways in a previous image. As shown inFIG. 23B, the airway tracking process receives as input a current imageat t₁ and airway information derived from a prior image, say prior imageat t₀. Based on the input, the system can determine an expected locationof the prior airway relative to the current image and determine thecurrent locations of the airways detected in the current image.

This process of identifying current locations of airways and expectedlocations of past airways is shown in FIG. 25. For example, in the firstrow, at time t₀ locations of airways A and B are identified, asrepresented in solid boxes. At time t₁ the temporal airway trackingreceives as input the detected locations of A and B and may optionallyestimate an updated location for these airways based on, for example,movement of the scope using, as way of example and not limitation,optical flow techniques. Examples of optical flow techniques may includemotion detection, object segmentation calculations, luminance, motioncompensated encoding, stereo disparity measurement, etc. The estimatedlocations of prior airways at time t₁ are shown in dashed boxes. Thetemporal airway tracking also analyzes the current image to detectlocations of airways detectable in the current image at time t₁. Thecurrent locations of the detected airways at the image at time t₁ isshown in solid boxes. FIG. 25 is described in greater detail below inconnection with Airway Association.

Accordingly, the airway tracking process determines, for a current time,the estimated locations of prior airways relative to the current imageand the locations of airways currently detected in the image data forthe current time.

E.2. Airway Association Based on Temporal and Spatial Analysis

In general, Airway Association, as executed by the system, may create arelationship between the airways detected in prior images and theairways detected in the current images. Airway Association creates theserelationships based on the locations of the prior airways and thelocations of the detected airways as may be determined by the airwaytracking. A relationship may represent cases where a prior airway is nowdetected as two separate airways. Such situations that may cause anairway to be depicted as two different airways include, but not belimited to: detecting airways visible through another airway (e.g.,descendent airways); and detecting a transition where an airwayinitially appears as a single airway but then become apparent that thesingle airway is actually multiple airways located in close proximitywith each other. Further, in some cases, an airway that is notoriginally part of a prior image may become visible and the AirwayAssociation may take actions to avoid relating that new airway withprior airways.

FIG. 24 illustrates examples of the type of relationship that can beexhibited by the airways in the previous and current images. As shown inFIG. 24, example relationships may include a one-to-one (A→A)association and a one-to-many (B→C,D) association illustrated in2400(a); a one-to-one (E→E) association and a many-to-one (F,G→H)association illustrated in 2400(b); a zero-to-one (none→J) and aone-to-one (I→I) association illustrated in 2400(c); and a one-to-zero(L→none) and a one-to-one (K→K) association illustrated in 2400(d). Thesolid rectangles are bounding boxes encapsulating the detected airwaysin the current image (e.g., image t₁), and the dashed rectangles arebounding boxes encapsulating the detected airways in the previous image(e.g., image t₀). Although rectangles and/or bounding boxes are used insome embodiments, such embodiments are not limited as such and may beextended to embodiments using any polygons or other shapes. Theassociations may be stored by the system as a series of records wherethe airways are given unique identifiers and may include data that linksthe unique identifiers for the airways that are associated together.

The system may determine a correspondence between the airways detectedin the previous image and the airways detected in the current image bycomparing the overlap between the rectangles (or more broadly, thelocation and shape of the airways) illustrated in FIG. 24. In someembodiments, one or more of the following overlap ratio values (or anycombination thereof) may be used to determine whether a given airwaycorresponds to another airway:

$\begin{matrix}{{O_{1}\left( {b_{1},b_{2}} \right)} = \frac{\Lambda \left( {b_{1}\bigcap b_{2}} \right)}{\min \left( {{\Lambda \left( b_{1} \right)},{\Lambda \left( b_{2} \right)}} \right)}} & (1) \\{{O_{2}\left( {b_{1},b_{2}} \right)} = \frac{\Lambda \left( {b_{1}\bigcap b_{2}} \right)}{\max \left( {{\Lambda \left( b_{1} \right)},{\Lambda \left( b_{2} \right)}} \right)}} & (2) \\{{O_{3}\left( {b_{1},b_{2}} \right)} = \frac{\Lambda \left( {b_{1}\bigcap b_{2}} \right)}{\Lambda \left( {b_{1}\bigcup b_{2}} \right)}} & (3)\end{matrix}$

where ∩ and ∪ denote the intersection and union operators, respectively,between two rectangles b₁ and b₂. The area (e.g., in pixels) of arectangle or other bounding shape (which may be calculated by countingthe pixels inside the rectangle) is denoted as Λ(⋅). min(p,q) andmax(p,q) return the smaller value and larger values from p and q,respectively.

In some embodiments, an association between two rectangles (e.g., onecorresponding to one or more airways detected in the previous image, andthe other corresponding to one or more airways detected in the currentimage) is established if the following condition is satisfied:

O(b ₁ ,b ₂)>τ  (4)

where τ is a pre-defined threshold and O(b₁, b₂) can be one of the threeoverlap ratio values (O₁, O₂ and O₃). The three overlap ratio values areexamples, and other overlap ratio values may be used instead of or inadditional to these overlap ratio values. Although rectangles are usedin the example of FIG. 24, in some embodiments, any geometric shapes(e.g., polygons or other shapes) may be used to determine the overlapbetween the airways. In such embodiments, the overlap between one set ofone or more geometric shapes corresponding to one or more airwaysdetected in the current picture and another set of one or more geometricshapes corresponding to one or more airways detected in the previouspicture may be calculated. In some embodiments, the geometric shapesused for determining the overlap fully encapsulate the detected airways.In other examples, the geometric shapes do not fully encapsulate thedetected airways and partially overlap with the airways. In the earlierexample in which the identity of the airways to be merged is known, thesystem may skip part or all of the geometrical overlap analysisillustrated in FIG. 24.

Although the above has been described as creating associations that areone level deep (e.g., one association between two or more airways), itis to be appreciated that other embodiments may create chains that aregreater than one level deep. For example, where airway X is associatedwith airways Y and Z, subsequent images may detect airways within Y,labelled Y′ and Y″, that meet the overlap threshold. In those cases, thesystem may then create a relationship between Y and Y′,Y″, such as anassociation chain that includes X→Y→Y′,Y″.

E.3. Adjusting the Overlap Threshold

FIG. 25 illustrates three examples in which incorrect associations maybe determined due to an incorrect setting of the overlap ratiothreshold. For example, the relationship shown in images 2500(a) shouldbe two one-to-one associations (e.g., A→A and B→B). However, the overlapillustrated in the fourth column of images 2500(a) may cause airway B ofthe current image to be associated with airway A of the previous image(or airways A,B to be associated with airways A,B of the previousimage). In such a case, the overlap threshold value may be adjusted sothat the overlap illustrated in the fourth column of images 2500(a) doesnot satisfy the overlap condition. Similarly images 2500(b) and 2500(c)illustrate other examples in which incorrect associations may bedetermined and how the overlap threshold value can be adjusted toprevent such associations (e.g., adjusted such that the overlapillustrated in the fourth column does not satisfy the overlapthreshold).

In some embodiments, the overlap ratio threshold can be a design-timeparameter configured by the designers of the system. However, in otherembodiments, the system may track and record surgical data to track thefailure rate of different overlap threshold values. Such systems mayutilize such surgical data to train the system on the use of differentoverlap threshold values and update the system with an updated overlapthreshold value that may lead to fewer failure rates. Furthermore,multiple systems may share the surgical data to increase the surgicaldata available for such training and selection of overlap thresholdvalues.

E.4. Airway Merging Based on Temporal and Spatial Analysis

In Airway Merging, the system may merge two or more current airwaysbased on the determined airway association(s) and data from preoperativemodel data. To determine whether to merge two or more airways in thecurrent image, the system may determine the current location of theinstrument and then determine the number of airways expected at thelocation from the preoperative model. It is to be appreciated that thesystem can utilize any number of approaches for determining a locationof the instrument. As way of example and not limitation, such approachesmay include the sensor fusion system described above, which may includeone or more of the following sensor types: EM, vision, robot data, shapesensing fiber, and the like.

The preoperative model data may be stored in a form that allows forretrieval of expected airways for a given location. Such data may begenerated before the procedure, where expected airways are indexed bythe location of the instrument with respect to the preoperative model.As used here, location may be a positional location with or withoutorientation, a segment with or without an insertion depth within thesegment, or any other representation of location.

Once the system determines that a set of airways in the current imageare to be merged, the system may determine the new airway center of themerged airway. The new airway centers can then be used in the imagebased airway mapping methods described above.

In the example of merging two airways having a single parent airway, theairway centers of the two airways, respectively, can be denoted as {x₁,y₁} and {x₂, y₂}. According to one approach, the airway center of themerged airway may be calculated as a center of the respective centerlocations of the airways detected in the current image (e.g., geometricshapes representative of the airways in the current image). For example,such a value may be calculated as follows:

$\begin{matrix}{\left\{ {x,y} \right\} = \left\{ {\frac{x_{1} + x_{2}}{2},\frac{y_{1} + y_{2}}{2}} \right\}} & (5)\end{matrix}$

In another example, the airway center of the merged airway may becalculated as a weighted center of the respective center locations ofthe airways detected in the current image (e.g., geometric shapesrepresentative of the airways in the current image). For example, such avalue may be calculated as follows:

$\begin{matrix}{\left\{ {x,y} \right\} = \left\{ {\frac{{x_{1} \cdot \Lambda_{2}} + {x_{2} \cdot \Lambda_{1}}}{\Lambda_{1} + \Lambda_{2}},\frac{{y_{1} \cdot \Lambda_{2}} + {y_{2} \cdot \Lambda_{1}}}{\Lambda_{1} + \Lambda_{2}}} \right\}} & (6)\end{matrix}$

where Λ₁ and Λ₂ are the areas (e.g., in pixels) of the two airways thatare being merged, respectively.

In yet another example, the airway center of the merged airway can becalculated as a center of a bounding polygon that encapsulates theairways detected in the current image (e.g., geometric shapesrepresentative of the airways in the current image). For example, if thecoordinates of the left-top corner and right-bottom corner of therectangle that encapsulates the first one of the two airways are denotedas {x₁ ^(LT), y₁ ^(LT)} and {x₁ ^(RB), y₁ ^(RB)} and the coordinates ofthe left-top corner and right-bottom corner of the rectangle thatencapsulates the second one of the two airways are denoted as {x₂ ^(LT),y₂ ^(LT)} and {x₂ ^(RB), y₂ ^(RB)} the corners {x^(LT), y^(LT)} and{x^(RB), y^(RB)} of the encapsulation rectangle can be derived asfollows:

x ^(LT)=min(x ₁ ^(LT) ,x ₂ ^(LT))  (7)

y ^(LT)=min(y ₁ ^(LT) ,y ₂ ^(LT))  (8)

x ^(RB)=max(x ₁ ^(RB) ,x ₂ ^(RB))  (9)

y ^(RB)=max(y ₁ ^(RB) ,y ₂ ^(RB))  (10)

Using these values, the airway center of the merged airway can becalculated as the center of the encapsulation rectangle, which isobtained as follows:

$\begin{matrix}{\left\{ {x,y} \right\} = \left\{ {\frac{x^{LT} + x^{RB}}{2},\frac{y^{LT} + y^{RB}}{2}} \right\}} & (11)\end{matrix}$

For example, the calculated merged center may be different from anyother center location of the airways detected in the current image. Asshown in the image 2200(b) of FIG. 22, a graphical representation of themerged center of the merged airway may be output on a display of thesystem instead of the centers of the airways that were merged into themerged airway. In some cases, the merged center may be output on thedisplay along with the centers of the airways that were merged. Althoughthe example techniques of calculating the airway center of the mergedairways are described above, other techniques of calculating the airwaycenter representative of the merged airways can be used.

F. Example Image-Based Airway Analysis and Mapping Methods and Systems

FIG. 26 illustrates an example method 2600 for implementing theimage-based airway analysis and mapping as described above. The method2600 can be implemented in various of the robotically-controlled systemsdescribed throughout this disclosure. The method 2600 can be implementedwith a robotic system including an instrument having an elongate bodyconfigured to be inserted into a luminal network. An imaging device canbe positioned on the elongate body (e.g., on a distal end of theelongate body). The instrument can be attached to an instrumentpositioning device (e.g., a robotic arm) configured to move theinstrument through the luminal network. A system employing the method2600 can include a processor configured with instructions that cause aprocessor to execute the method 2600. The method 2600 is provided by wayof example only and the image-based airway analysis and mapping can beimplemented using different steps than those shown in FIG. 26. Forsimplicity, the steps illustrated in FIG. 26 are described as beingperformed by a system (e.g., one of the systems described herein or asystem configured to perform one or more of the techniques describedhere).

At block 2602, the system for mapping one or more airways in a luminalnetwork captures a plurality of images within the luminal network withan imaging device positioned on an instrument. The plurality of imagesmay comprise at least two images, where one of the images (e.g., currentimage, or second image) is captured at a time subsequent to the otherimage (e.g., previous image, or first image). The time at which thefirst image is captured may be referred to as a first time, and the timeat which the second image is captured may be referred to as a secondtime.

At block 2604, the system identifies one or more airways in the firstimage. In some embodiments, the system identifies a single airway in thefirst image. In other embodiments, the system identifies multipleairways in the first image. The system may utilize any of the airwayidentification or detection techniques described herein (e.g., pixelintensity analysis) or other techniques. For example, the identifiedairways may include one or more polygons or other geometrical shapeswhose pixels satisfy the pixel intensity threshold specified by theairway detection method. In some embodiments, a point location such as acentroid is determined and associated with the detected airway.

At block 2606, the system the system identifies two or more airways inthe second image. In some embodiments, the system identifies a singleairway in the second image. In other embodiments, the system identifiesmultiple airways in the second image. The system may identify the one ormore airways in the second image in a manner similar to that utilized atblock 2604.

At block 2608, the system determines, based on the first airway in thefirst image and the two or more airways in the second image, that anoverlap condition is met. The determination that the overlap conditionis met may involve, for example: determining a degree of spatial overlapbetween a first set of one or more geometric shapes representative ofthe first airway in the first image and a second set of one or moregeometric shapes representative of the two or more airways in the secondimage; and determining the degree of spatial overlap meets or exceeds adefined overlap threshold value, parameter, or indicia. It is to beappreciated that blocks 2602-2606 may involve one or more featuresdescribed with respect to Airway Tracking and Airway Association. Thatis, the system may track the location and sizes of airways in a currentimage as well as the expected locations and sizes of airways previouslydetected in prior images. When an overlap condition is met, the systemmay create an association with airways in a prior image and one or moreimages in a current image. Other associations, as explained above, maybe one-to-one or many-to-one.

At block 2610, the system accesses preoperative model data indicative ofan expected count of airways corresponding to a location of theinstrument during the second time. For example, although not illustratedin FIG. 26, the system may access a position state estimate for theinstrument positioned within the luminal network. The position stateestimate can include an identification of which branch the instrument iscurrently positioned (e.g., in the hierarchical structure of luminalnetwork 1000 of FIG. 21). Based on the position state estimate and thepreoperative model data (e.g., CT images), the system may determine theexpected count of airways corresponding to the location of theinstrument during the second time (i.e. at the time the second image iscaptured). The position state estimate can be determined, for example,by the navigation and localization system 90 of FIG. 15. The positionstate estimate can be determined based on various and/or multipleposition sensing modalities and information, such as preoperative modeldata 91, vision data 92, EM data 93 (or other position sensing data),and/or robotic command and kinematics data 94.

At block 2612, the system determines, based on the preoperative modeldata and the determination that the overlap condition is met, a mappingbetween the two or more airways in the second image and the airways inthe preoperative model data. At this block, the system may determine themapping between airways in the second image and the airways in thepreoperative model data using Airway Merging (see section E.4.). Thatis, according to some embodiments, the system may use the data derivedfrom blocks 2602-2606 (Airway Tracking and Airway Association) and acount of the expected number of airways for the current location withinthe luminal network, as may be determined by preoperative model data, todetermine whether or not to merge airways detected in the second image.The system may then map the merged or non-merged to airways in thepreoperative model. Mapping airways to preoperative model data isfurther described in U.S. patent application Ser. No. 15/783,903, filedon Oct. 13, 2017, the entirety of which is incorporated herein byreference and appended hereto as Appendix A.

As just discussed, as part of block 2612, the system may determine themapping between the two or more airways in the second image and theairways in the preoperative model data based on the preoperative modelby accessing the preoperative model data indicative of an expected countof airways corresponding to a location of the instrument during thesecond time. Accessing the preoperative model data to determine expectedairway counts is now explained in greater detail. Although notillustrated in FIG. 26, the preoperative model data may include datastructures that allow the system to retrieve the number of airwaysexpected for a given location. Further, the system may access a positionstate estimate for the instrument positioned within the luminal network.The position state estimate can include an identification of whichbranch the instrument is currently positioned (e.g., in the hierarchicalstructure of luminal network 1000 of FIG. 21). Based on the positionstate estimate and the preoperative model data (e.g., data derived fromCT images, in one embodiment), the system may determine the expectedcount of airways corresponding to the location of the instrument duringthe second time (i.e. at the time the second image is captured) byaccessing the aforementioned data structures. The position stateestimate can be determined, for example, by the navigation andlocalization system 90 of FIG. 15. The position state estimate can bedetermined based on various and/or multiple position sensing modalitiesand information, such as preoperative model data 91, vision data 92, EMdata 93 (or other position sensing data), and/or robotic command andkinematics data 94.

As further examples, the preoperative model data may indicate that thesystem is expected to detect branches 1004, 1006 of the luminal network1000. If the system has detected two airways at block 2606, the systemmay determine that merging is not needed based on the count of the twoor more airways detected in the second image being equal to the expectedcount of airways indicated by the preoperative model data. In such acase, the system may map the two detected airways to the branches 1004,1006, respectively (e.g., based on the orientation of the image and/orthe two airways) shown in FIG. 21. In the example of FIG. 26, based onthe determination that the count of the two or more airways identifiedin the second image is equal to the expected count of airways indicatedby the preoperative model data, the system, at block 2612, may determinethe mapping such that each one of the two or more airways identified inthe second image is mapped to a different one of the airways in thepreoperative model data.

In another example, as illustrated in FIG. 22, the preoperative modeldata may indicate that the system is expected to detect branches 1004,1006 of the luminal network 1000, but the system may have detected fourairways at block 2606 (e.g., branches 1004, 1012, 1014, 1016). In such acase, the system may merge the four airways into two airways (e.g., bymerging the detected airways corresponding to the branches 1012, 1014,1016 into a single airway), and map the merged airway to the branch1006, and the unmerged airway to the branch 1004. In the example of FIG.26, based on the determination that the count of the two or more airwaysidentified in the second image is different than the expected count ofairways indicated by the preoperative model data, the system, at block2612, may determine the mapping such that at least two of the two ormore airways identified in the second image is mapped to a single airwayin the preoperative model data. In some embodiments, mapping comprisesstoring the association between the two or more airways in a database.For example, the system may store an indication that a given airwayidentified in the image branches into two or more airways indicated bythe preoperative model data.

As discussed above, the system may confirm, based on the mapping betweenthe two or more airways in the second image and the airways in thepreoperative model data, a segment of the luminal network that theinstrument is currently located. In some embodiments, the system mayadjust, based on the mapping between the two or more airways in thesecond image and the airways in the preoperative model data, aconfidence value of a position state estimate based on other data. Forexample, this other data may include EM data, robotic data, opticalshape sensor data, and inertial measurement unit (IMU) data, any otherinput data described herein (e.g., input data 91-94 of FIG. 15), or anycombination thereof. Additionally or alternatively, the system maypredict, based on the mapping between the two or more airways in thesecond image and the airways in the preoperative model data, a segmentof the luminal network that the instrument will be entering.

In one embodiment of block 2608, upon determining that a combination ofairways in the current image overlaps with a combination of airways inthe previous image by more than a threshold amount (or equal to thethreshold amount), the system may merge the combination of airways inthe current image into one or more airways such that the number ofairways in the current image after the merger matches the number ofairways in the combination of airways in the previous image. Forexample, the system may compare an overlap ratio value (e.g., any of theoverlap ratio values in Equations (1)-(3) above) to a threshold overlapvalue to determine whether an association between the two combinationsof airways exists.

As discussed above, the system may also determine the merged center ofthe merged airway. For example, the system may calculate a merged centerlocation of the combination of airways in the current image (e.g., theset of geometric shapes representative of the combination of airways inthe current image), where the merged center location is different thanany of respective center locations of the airways detected in thecurrent image (e.g., the geometric shapes in the set that arerepresentative of the combination of airways in the current image).

In the case that the system determines that multiple pairings ofcombinations of airways satisfy the overlap condition (e.g., A→B having60% overlap and A→C having 25% overlap), the system may determine thepairing of combinations of airways that has a greater overlap to be inassociation (e.g., A→B). In some embodiments, the determination ofwhether an overlap condition is satisfied is not one of degree (e.g.,overlap ratio being greater than or equal to a threshold overlap ratio),and is instead a binary determination (e.g., whether the mid-point ofthree or more segments of a rectangle encapsulating the combination ofairways in the current image is within a rectangle encapsulating thecombination of airways in the parent image).

In some embodiments, the determination of whether the overlap conditionis met may include determining a bounding polygon (e.g., rectangle orother shapes) that encapsulates the combination of airways in theprevious image (e.g., the set of geometric shapes representative of thecombination of airways in the previous image), determining a boundingpolygon that encapsulates the combination of airways in the currentimage (e.g., the set of geometric shapes representative of thecombination of airways in the current image), and determining the degreeof spatial overlap based on an overlap between the two boundingpolygons.

The image-based airway analysis and mapping techniques described hereincompensate the Vision Branch Prediction (VBP) approach or otherimage-based branch prediction approaches. VBP may not able to performcorrect airway mapping in some cases due to the visibility of airwaysthat are from different generations in a given image (e.g., asillustrated in image 2200(a) of FIG. 22). The image-based airwayanalysis and mapping techniques described herein can determine orconfirm the hierarchical structure of the airways in a luminal networkand determine airway centers that correspond to the same generation.This helps establish correct airway mappings, as illustrated in image2200(b) of FIG. 22).

G. Recursive Distance Searching

In some embodiments, upon determining that two or more airways in thecurrent image are to be merged, the system can merge the airways usingrecursive distance searching. In doing so, the system may assume that anairway is closer to the airways from its same generation than othergenerations. For example, this assumption holds true in the example ofFIGS. 21 and 22, where the airways in the third generation (e.g.,branches 1012, 1014, 1016, illustrated in the bottom right corner inFIG. 22) are all closer to each other than the airway in the secondgeneration (e.g., branch 1004, illustrated in the top left corner inFIG. 22).

In the recursive distance searching may involve a set P={p_(i)}_(i=1)^(n) which represents a set of n airway centers detected in the currentimage. Each airway center may be denoted as p_(i)={x_(i), y_(i)}. Then,the system may compute the following value:

$\begin{matrix}{\left\{ {p_{i},p_{j}} \right\} = {\underset{\underset{i < j}{p_{i},{p_{j} \in P}}}{\arg \mspace{11mu} \min}{{p_{i} - p_{j}}}}} & (12)\end{matrix}$

where ∥⋅∥ denotes the L2 norm operator. The above equation finds thepair of centers that have the closest distance to each other among theairways centers in the set P, and the system may assume that these twoairways are from the same generation. The system then merges the twoairways (e.g., obtains the airway center of the merged airway), forexample, based on Equations (5)-(11). The system then removes p_(i) andp_(j) from P and add p_(k) into P and repeats this process (e.g.,computation, merger, removal, and addition) until only one airway (or adesired number of airways) remains in P.

IV. Implementing Systems and Terminology

Implementations disclosed herein provide systems, methods and apparatusfor image-based airway analysis and mapping for navigationrobotically-controlled medical instruments. Various implementationsdescribed herein provide for improved navigation of luminal networks.

It should be noted that the terms “couple,” “coupling,” “coupled” orother variations of the word couple as used herein may indicate eitheran indirect connection or a direct connection. For example, if a firstcomponent is “coupled” to a second component, the first component may beeither indirectly connected to the second component via anothercomponent or directly connected to the second component.

The position estimation and robotic motion actuation functions describedherein may be stored as one or more instructions on a processor-readableor computer-readable medium. The term “computer-readable medium” refersto any available medium that can be accessed by a computer or processor.By way of example, and not limitation, such a medium may comprise randomaccess memory (RAM), read-only memory (ROM), electrically erasableprogrammable read-only memory (EEPROM), flash memory, compact discread-only memory (CD-ROM) or other optical disk storage, magnetic diskstorage or other magnetic storage devices, or any other medium that canbe used to store desired program code in the form of instructions ordata structures and that can be accessed by a computer. It should benoted that a computer-readable medium may be tangible andnon-transitory. As used herein, the term “code” may refer to software,instructions, code or data that is/are executable by a computing deviceor processor.

The methods disclosed herein comprise one or more steps or actions forachieving the described method. The method steps and/or actions may beinterchanged with one another without departing from the scope of theclaims. In other words, unless a specific order of steps or actions isrequired for proper operation of the method that is being described, theorder and/or use of specific steps and/or actions may be modifiedwithout departing from the scope of the claims.

As used herein, the term “plurality” denotes two or more. For example, aplurality of components indicates two or more components. The term“determining” encompasses a wide variety of actions and, therefore,“determining” can include calculating, computing, processing, deriving,investigating, looking up (e.g., looking up in a table, a database oranother data structure), ascertaining and the like. Also, “determining”can include receiving (e.g., receiving information), accessing (e.g.,accessing data in a memory) and the like. Also, “determining” caninclude resolving, selecting, choosing, establishing and the like.

The phrase “based on” does not mean “based only on,” unless expresslyspecified otherwise. In other words, the phrase “based on” describesboth “based only on” and “based at least on.”

The previous description of the disclosed implementations is provided toenable any person skilled in the art to make or use the presentinvention. Various modifications to these implementations will bereadily apparent to those skilled in the art, and the generic principlesdefined herein may be applied to other implementations without departingfrom the scope of the invention. For example, it will be appreciatedthat one of ordinary skill in the art will be able to employ a numbercorresponding alternative and equivalent structural details, such asequivalent ways of fastening, mounting, coupling, or engaging toolcomponents, equivalent mechanisms for producing particular actuationmotions, and equivalent mechanisms for delivering electrical energy.Thus, the present invention is not intended to be limited to theimplementations shown herein but is to be accorded the widest scopeconsistent with the principles and novel features disclosed herein.

What is claimed is:
 1. A method of navigating an instrument through aluminal network, the method comprising: capturing a plurality of imageswithin the luminal network with an imaging device positioned on theinstrument, the plurality of images comprising at least a first imagecaptured at a first time and a second image captured at a second timesubsequent to the first time; identifying a first airway in the firstimage; identifying two or more airways in the second image; determining,based on the first airway in the first image and the two or more airwaysin the second image, that an overlap condition is met; accessingpreoperative model data indicative of an expected count of airwayscorresponding to a location of the instrument during the second time;and determining, based on the preoperative model data and thedetermination that the overlap condition is met, a mapping between thetwo or more airways in the second image and the airways in thepreoperative model data.
 2. The method of claim 1, wherein determiningthe mapping comprises: determining that a count of the two or moreairways identified in the second image is different than the expectedcount of airways indicated by the preoperative model data; anddetermining, based on the determination that the overlap condition ismet, that the two or more airways branch from the first airway.
 3. Themethod of claim 1, wherein determining that the overlap condition is metis further based on a degree of spatial overlap between a first set ofone or more geometric shapes representative of the first airway in thefirst image and a second set of one or more geometric shapesrepresentative of the two or more airways in the second image.
 4. Themethod of claim 3, further comprising: determining a first boundingpolygon that encapsulates the first set of geometric shapesrepresentative of the first airway in the first image; and determining asecond bounding polygon that encapsulates the second set of geometricshapes representative of the two or more airways in the second image,wherein determining the degree of spatial overlap is based on an overlapbetween the first bounding polygon and the second bounding polygon. 5.The method of claim 3, wherein determining that the overlap condition ismet is further based on a determination that the degree of spatialoverlap between the first set of one or more geometric shapes and thesecond set of one or more geometric shapes is greater than a thresholdoverlap amount.
 6. The method of claim 3, wherein: determining themapping comprises calculating a merged center location of the second setof geometric shapes representative of the two or more airways identifiedin the second image, and the merged center location is different thanany of respective center locations of the geometric shapes in the secondset.
 7. The method of claim 6, wherein the merged center location is oneof (i) a center of the respective center locations of the geometricshapes in the second set, (ii) a weighted center of the respectivecenter locations of the geometric shapes in the second set, or (iii) acenter of a bounding polygon that encapsulates the geometric shapes inthe second set.
 8. The method of claim 6, further comprising causing agraphical indication to be presented over the second image at the centerlocation of the merged airway.
 9. The method of claim 1, furthercomprising predicting, based on the mapping between the two or moreairways in the second image and the airways in the preoperative modeldata, a segment of the luminal network that the instrument will beentering.
 10. The method of claim 1, further comprising confirming,based on the mapping between the two or more airways in the second imageand the airways in the preoperative model data, a segment of the luminalnetwork that the instrument is currently located.
 11. A non-transitorycomputer readable storage medium having stored thereon instructionsthat, when executed, cause a processor of a device to at least: capturea plurality of images within a luminal network with an imaging devicepositioned on an instrument, the plurality of images comprising at leasta first image captured at a first time and a second image captured at asecond time subsequent to the first time; identify a first airway in thefirst image; identify two or more airways in the second image;determine, based on the first airway in the first image and the two ormore airways in the second image, that an overlap condition is met;access preoperative model data indicative of an expected count ofairways corresponding to a location of the instrument during the secondtime; and determine, based on the preoperative model data and thedetermination that the overlap condition is met, a mapping between thetwo or more airways in the second image and the airways in thepreoperative model data.
 12. The non-transitory computer readablestorage medium of claim 11, wherein determining the mapping comprises:determining that a count of the two or more airways identified in thesecond image is different than the expected count of airways indicatedby the preoperative model data; and determining, based on thedetermination that the overlap condition is met, that the two or moreairways branch from the first airway.
 13. The non-transitory computerreadable storage medium of claim 11, wherein determining that theoverlap condition is met is further based on a degree of spatial overlapbetween a first set of one or more geometric shapes representative ofthe first airway in the first image and a second set of one or moregeometric shapes representative of the two or more airways in the secondimage.
 14. The non-transitory computer readable storage medium of claim11, wherein the instructions, when executed, further cause the processorto predict, based on the mapping between the two or more airways in thesecond image and the airways in the preoperative model data, a segmentof the luminal network that the instrument will be entering.
 15. Thenon-transitory computer readable storage medium of claim 11, wherein theinstructions, when executed, further cause the processor to confirm,based on the mapping between the two or more airways in the second imageand the airways in the preoperative model data, a segment of the luminalnetwork that the instrument is currently located.
 16. The non-transitorycomputer readable storage medium of claim 11, wherein the instructions,when executed, further cause the processor to adjust, based on themapping between the two or more airways in the second image and theairways in the preoperative model data, a confidence value of a positionstate estimate based on data selected from the group consisting ofelectromagnetic (EM) data, robotic data, optical shape sensor data, andinertial measurement unit (IMU) data.
 17. The non-transitory computerreadable storage medium of claim 11, wherein the one or more geometricshapes in the first set comprise one or more polygons that at leastpartially overlap with the first airway in the first image, and the oneor more geometric shapes in the second set comprise one or more polygonsthat at least partially overlap with the two or more airways in thesecond image.
 18. The non-transitory computer readable storage medium ofclaim 11, wherein determining the mapping comprises: determining that acount of the two or more airways identified in the second image is equalto the expected count of airways indicated by the preoperative modeldata; and determining the mapping such that each one of the two or moreairways identified in the second image is mapped to a different one ofthe airways in the preoperative model data in response to determiningthat the count of the two or more airways identified in the second imageis equal to the expected count of airways indicated by the preoperativemodel data.
 19. The non-transitory computer readable storage medium ofclaim 11, wherein determining the mapping comprises: determining that acount of the two or more airways identified in the second image isdifferent than the expected count of airways indicated by thepreoperative model data; and determining the mapping such that at leasttwo of the two or more airways identified in the second image is mappedto a single airway in the preoperative model data in response todetermining that the count of the two or more airways identified in thesecond image is different than the expected count of airways indicatedby the preoperative model data.
 20. A robotic surgical system formapping one or more airways in a luminal network, the system comprising:an instrument having: an elongate body configured to be inserted intothe luminal network, and an imaging device positioned on a distalportion of the elongate body; an instrument positioning device attachedto the instrument, the instrument positioning device configured to movethe instrument through the luminal network; at least onecomputer-readable memory having stored thereon executable instructions;and one or more processors in communication with the at least onecomputer-readable memory and configured to execute the instructions tocause the system to at least: capture a plurality of images within theluminal network with an imaging device positioned on the instrument, theplurality of images comprising at least a first image captured at afirst time and a second image captured at a second time subsequent tothe first time; identify a first airway in the first image; identify twoor more airways in the second image; determine, based on the firstairway in the first image and the two or more airways in the secondimage, that an overlap condition is met; access preoperative model dataindicative of an expected count of airways corresponding to a locationof the instrument during the second time; and determine, based on thepreoperative model data and the determination that the overlap conditionis met, a mapping between the two or more airways in the second imageand the airways in the preoperative model data.
 21. The robotic surgicalsystem of claim 20, wherein determining that the overlap condition ismet is further based on a degree of spatial overlap between a first setof one or more geometric shapes representative of the first airway inthe first image and a second set of one or more geometric shapesrepresentative of the two or more airways in the second image.
 22. Therobotic surgical system of claim 21, wherein the one or more processorsare further configured to: determine a first bounding polygon thatencapsulates the first set of geometric shapes representative of thefirst airway in the first image; and determine a second bounding polygonthat encapsulates the second set of geometric shapes representative ofthe two or more airways in the second image, wherein determining thedegree of spatial overlap is based on an overlap between the firstbounding polygon and the second bounding polygon.
 23. The roboticsurgical system of claim 21, wherein determining that the overlapcondition is met is further based on a determination that the degree ofspatial overlap between the first set of one or more geometric shapesand the second set of one or more geometric shapes is greater than athreshold overlap amount.
 24. The robotic surgical system of claim 21,wherein determining the mapping comprises calculating a merged centerlocation of the second set of geometric shapes representative of the twoor more airways identified in the second image, and the merged centerlocation is different than any of respective center locations of thegeometric shapes in the second set.
 25. The robotic surgical system ofclaim 20, wherein the one or more processors are further configured topredict, based on the mapping between the two or more airways in thesecond image and the airways in the preoperative model data, a segmentof the luminal network that the instrument will be entering.
 26. Therobotic surgical system of claim 20, wherein the one or more processorsare further configured to confirm, based on the mapping between the twoor more airways in the second image and the airways in the preoperativemodel data, a segment of the luminal network that the instrument iscurrently located.
 27. The robotic surgical system of claim 20, whereinthe one or more processors are further configured to adjust, based onthe mapping between the two or more airways in the second image and theairways in the preoperative model data, a confidence value of a positionstate estimate based on data selected from the group consisting ofelectromagnetic (EM) data, robotic data, optical shape sensor data, andinertial measurement unit (IMU) data.
 28. The robotic surgical system ofclaim 20, wherein the one or more geometric shapes in the first setcomprise one or more polygons that at least partially overlap with thefirst airway in the first image, and the one or more geometric shapes inthe second set comprise one or more polygons that at least partiallyoverlap with the two or more airways in the second image.
 29. Therobotic surgical system of claim 20, wherein determining the mappingcomprises: determining that a count of the two or more airwaysidentified in the second image is equal to the expected count of airwaysindicated by the preoperative model data; and determining the mappingsuch that each one of the two or more airways identified in the secondimage is mapped to a different one of the airways in the preoperativemodel data in response to determining that the count of the two or moreairways identified in the second image is equal to the expected count ofairways indicated by the preoperative model data.
 30. The roboticsurgical system of claim 20, wherein determining the mapping comprises:determining that a count of the two or more airways identified in thesecond image is different than the expected count of airways indicatedby the preoperative model data; and determining the mapping such that atleast two of the two or more airways identified in the second image ismapped to a single airway in the preoperative model data in response todetermining that the count of the two or more airways identified in thesecond image is different than the expected count of airways indicatedby the preoperative model data.